[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:10:17.860915 23886 test_util.cc:268] Using random seed: 198461216
I20240501 14:10:17.867376 23886 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:10:17.867429 23886 ts_itest-base.cc:113] --------------
I20240501 14:10:17.867439 23886 ts_itest-base.cc:114] 6 tablet servers
I20240501 14:10:17.867449 23886 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:10:17.867457 23886 ts_itest-base.cc:116] --------------
2024-05-01T14:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:10:17Z Disabled control of system clock
I20240501 14:10:17.876720 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.83.190:33869
--webserver_interface=127.23.83.190
--webserver_port=0
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.83.190:33869
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=40 with env {}
I20240501 14:10:18.052184 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 23898
I20240501 14:10:18.052331 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:10:18.108660 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.129:0
--local_ip_for_outbound_sockets=127.23.83.129
--webserver_interface=127.23.83.129
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:18.296087 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 23989
I20240501 14:10:18.296196 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:10:18.298477 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.130:0
--local_ip_for_outbound_sockets=127.23.83.130
--webserver_interface=127.23.83.130
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:18.464784 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 24120
I20240501 14:10:18.464901 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:10:18.467327 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.131:0
--local_ip_for_outbound_sockets=127.23.83.131
--webserver_interface=127.23.83.131
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:18.643761 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 24251
I20240501 14:10:18.643870 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:10:18.646248 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.132:0
--local_ip_for_outbound_sockets=127.23.83.132
--webserver_interface=127.23.83.132
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:18.832274 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 24382
I20240501 14:10:18.832368 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:10:18.834496 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.133:0
--local_ip_for_outbound_sockets=127.23.83.133
--webserver_interface=127.23.83.133
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:19.000090 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 24513
I20240501 14:10:19.000185 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:10:19.003629 23886 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWs8th6/build/debug/bin/kudu
/tmp/dist-test-taskWs8th6/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.83.134:0
--local_ip_for_outbound_sockets=127.23.83.134
--webserver_interface=127.23.83.134
--webserver_port=0
--tserver_master_addrs=127.23.83.190:33869
--builtin_ntp_servers=127.23.83.148:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 14:10:19.179802 23886 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWs8th6/build/debug/bin/kudu as pid 24644
I20240501 14:10:19.179898 23886 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:10:19.182427 23886 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:10:19.262178 23886 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:10:19.264261 23886 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 86e43abaefd642c3ba7d98bd3bae8b21 to finish bootstrapping
I20240501 14:10:19.270314 23886 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1ebfab2167974c67908ba74179b736be to finish bootstrapping
I20240501 14:10:19.276465 23886 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c8e924dcdb9d43c7916f608ebbf71dcb to finish bootstrapping
I20240501 14:10:19.282047 23886 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f3cbf2729d8a45c3bbd222e800d2fffb to finish bootstrapping
I20240501 14:10:19.287225 23886 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 77a65181cc774449b32220b150f27ea1 to finish bootstrapping
I20240501 14:10:19.293087 23886 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 601fcf5e368d40288a9db33f7b4e060f to finish bootstrapping
I20240501 14:10:19.299000 23886 test_util.cc:268] Using random seed: 199899310
F20240501 14:10:28.920068 25345 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.queue_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.queue_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_cpu_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_cpu_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_wall_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_wall_time_us) (tablet-download-pool-183584db7f7a4468aa534a0a334280ae.queue_time_us, tablet-download-pool-183584db7f7a4468aa534a0a334280ae.queue_time_us) (tablet-download-pool-183584db7f7a4468aa534a0a334280ae.run_cpu_time_us, tablet-download-pool-183584db7f7a4468aa534a0a334280ae.run_cpu_time_us) (tablet-download-pool-183584db7f7a4468aa534a0a334280ae.run_wall_time_us, tablet-download-pool-183584db7f7a4468aa534a0a334280ae.run_wall_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.queue_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.queue_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.run_cpu_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.run_cpu_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.run_wall_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.run_wall_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.queue_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.queue_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_cpu_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_cpu_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_wall_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_wall_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.queue_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.queue_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.run_cpu_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.run_cpu_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.run_wall_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.run_wall_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.queue_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.queue_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_cpu_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_cpu_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_wall_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_wall_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_cpu_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_cpu_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_wall_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_wall_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.queue_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.queue_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_cpu_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_cpu_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_wall_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f773a55adcd google::LogMessage::Fail() at ??:0
@ 0x7f773a55eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f773a55a7cc google::LogMessage::Flush() at ??:0
@ 0x7f773a55bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f773b9c4044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f773b9a8f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f773b9a6efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f773e55977a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f773e562a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f773e5e376f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f773e5f74f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f773e5e2226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f773e5f1c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55920ba8f75c std::function<>::operator()() at ??:0
@ 0x7f773b9abe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f773b9ac6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f773b9ae03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55920ba8f75c std::function<>::operator()() at ??:0
@ 0x7f773b99b807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f773c1d96db start_thread at ??:0
@ 0x7f773971171f clone at ??:0
W20240501 14:10:29.422441 24807 meta_cache.cc:302] tablet 06f46b6599104821b6446e5c5d94d2d8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | HEALTHY | <none> | 5 | 0
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | HEALTHY | <none> | 11 | 0
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 0 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 9 | 0
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | HEALTHY | <none> | 10 | 0
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.130 | experimental | 127.23.83.130:36363
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.132 | experimental | 127.23.83.132:44579
local_ip_for_outbound_sockets | 127.23.83.134 | experimental | 127.23.83.134:44847
never_fsync | true | unsafe,advanced | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.23.83.130:36363
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.83.132:44579
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.83.134:44847
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
time_source | builtin | 127.23.83.130:36363, 127.23.83.134:44847, 127.23.83.129:36771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.134:44847, tserver@127.23.83.129:36771, and 3 other server(s)
Tablet Summary
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = f3cbf2729d8a45c3bbd222e800d2fffb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
D = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A B C D~ | 3 | 58 | Yes
B | A* B C D~ | 2 | 58 | Yes
C | A* B C D~ | 2 | 58 | Yes
D | [config not available] | | |
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 67 | Yes
B | A B C | 3 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 64 | Yes
B | A B C* | 3 | 64 | Yes
C | A B* C | 2 | 64 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable [NONVOTER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
D = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 2 | 60 | Yes
B | A B C D~ | 3 | 60 | Yes
C | A B* C D~ | 2 | 60 | Yes
D | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | A* B C | 1 | 65 | Yes
C | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 67 | Yes
B | A* B C | 1 | 67 | Yes
C | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 60 | Yes
B | A* B C | 1 | 60 | Yes
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | A* B C | 1 | 65 | Yes
C | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable [NONVOTER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
D = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 2 | 62 | Yes
B | A B C* | 2 | 62 | Yes
C | A B C* | 2 | 62 | Yes
D | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 61 | Yes
B | A B* C | 1 | 61 | Yes
C | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable [NONVOTER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
D = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 2 | 65 | Yes
B | A* B C D~ | 3 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 61 | Yes
B | A B* C | 1 | 61 | Yes
C | [config not available] | | |
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 60 | Yes
B | A* B C | 1 | 60 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable [NONVOTER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
D = 77a65181cc774449b32220b150f27ea1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 2 | 55 | Yes
B | A B C D~ | 3 | 55 | Yes
C | A B* C D~ | 2 | 55 | Yes
D | [config not available] | | |
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 40 | 6 | 0 | 27 | 7
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 20
Median | 21
Third Quartile | 21
Maximum | 33
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | 77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | 33
Small | 86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | 13
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:10:33.022547 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
F20240501 14:10:36.151835 26195 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-068eeaab02144dec9e09f85852476dc2.queue_time_us, tablet-download-pool-068eeaab02144dec9e09f85852476dc2.queue_time_us) (tablet-download-pool-068eeaab02144dec9e09f85852476dc2.run_cpu_time_us, tablet-download-pool-068eeaab02144dec9e09f85852476dc2.run_cpu_time_us) (tablet-download-pool-068eeaab02144dec9e09f85852476dc2.run_wall_time_us, tablet-download-pool-068eeaab02144dec9e09f85852476dc2.run_wall_time_us) (tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.queue_time_us, tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.queue_time_us) (tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.run_cpu_time_us, tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.run_cpu_time_us) (tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.run_wall_time_us, tablet-download-pool-06f46b6599104821b6446e5c5d94d2d8.run_wall_time_us) (tablet-download-pool-12cab77f880d490b8a72d56a0629efe8.queue_time_us, tablet-download-pool-12cab77f880d490b8a72d56a0629efe8.queue_time_us) (tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.queue_time_us, tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.queue_time_us) (tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.run_cpu_time_us, tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.run_cpu_time_us) (tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.run_wall_time_us, tablet-download-pool-14dad87a8a604796afb5f6ca72adee32.run_wall_time_us) (tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.queue_time_us, tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.queue_time_us) (tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.run_cpu_time_us, tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.run_cpu_time_us) (tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.run_wall_time_us, tablet-download-pool-1ce14c9f93814647b255fae7853cf8bf.run_wall_time_us) (tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.queue_time_us, tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.queue_time_us) (tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.run_cpu_time_us, tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.run_cpu_time_us) (tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.run_wall_time_us, tablet-download-pool-2eccecd65b8c4ad49eb3414afd96ab90.run_wall_time_us) (tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.queue_time_us, tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.queue_time_us) (tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.run_cpu_time_us, tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.run_cpu_time_us) (tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.run_wall_time_us, tablet-download-pool-33edb7a66a7a4f2c8f397a521da5fda2.run_wall_time_us) (tablet-download-pool-402e8cd189774227941101fd68782a89.queue_time_us, tablet-download-pool-402e8cd189774227941101fd68782a89.queue_time_us) (tablet-download-pool-402e8cd189774227941101fd68782a89.run_cpu_time_us, tablet-download-pool-402e8cd189774227941101fd68782a89.run_cpu_time_us) (tablet-download-pool-402e8cd189774227941101fd68782a89.run_wall_time_us, tablet-download-pool-402e8cd189774227941101fd68782a89.run_wall_time_us) (tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.queue_time_us, tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.queue_time_us) (tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.run_cpu_time_us, tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.run_cpu_time_us) (tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.run_wall_time_us, tablet-download-pool-4248d5ae85e34b969fa59c0320c8eae4.run_wall_time_us) (tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.queue_time_us, tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.queue_time_us) (tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.run_cpu_time_us, tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.run_cpu_time_us) (tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.run_wall_time_us, tablet-download-pool-7153494f545a4a12a9650e1f5f58c1ed.run_wall_time_us) (tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.queue_time_us, tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.queue_time_us) (tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.run_cpu_time_us, tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.run_cpu_time_us) (tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.run_wall_time_us, tablet-download-pool-7d42976bfa624cdd82cf06b6e2e41c8d.run_wall_time_us) (tablet-download-pool-9a8785261879441d998a669efd4f06f3.queue_time_us, tablet-download-pool-9a8785261879441d998a669efd4f06f3.queue_time_us) (tablet-download-pool-9a8785261879441d998a669efd4f06f3.run_cpu_time_us, tablet-download-pool-9a8785261879441d998a669efd4f06f3.run_cpu_time_us) (tablet-download-pool-9a8785261879441d998a669efd4f06f3.run_wall_time_us, tablet-download-pool-9a8785261879441d998a669efd4f06f3.run_wall_time_us) (tablet-download-pool-a915c02a73cf448091632815ff49557f.queue_time_us, tablet-download-pool-a915c02a73cf448091632815ff49557f.queue_time_us) (tablet-download-pool-a915c02a73cf448091632815ff49557f.run_cpu_time_us, tablet-download-pool-a915c02a73cf448091632815ff49557f.run_cpu_time_us) (tablet-download-pool-a915c02a73cf448091632815ff49557f.run_wall_time_us, tablet-download-pool-a915c02a73cf448091632815ff49557f.run_wall_time_us) (tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.queue_time_us, tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.queue_time_us) (tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.run_cpu_time_us, tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.run_cpu_time_us) (tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.run_wall_time_us, tablet-download-pool-bbff0f0cb864400f9268f92a6c6cca6f.run_wall_time_us) (tablet-download-pool-d0a640c883a14d1e8715a534595a6877.queue_time_us, tablet-download-pool-d0a640c883a14d1e8715a534595a6877.queue_time_us) (tablet-download-pool-d0a640c883a14d1e8715a534595a6877.run_cpu_time_us, tablet-download-pool-d0a640c883a14d1e8715a534595a6877.run_cpu_time_us) (tablet-download-pool-d0a640c883a14d1e8715a534595a6877.run_wall_time_us, tablet-download-pool-d0a640c883a14d1e8715a534595a6877.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f087b03ddcd google::LogMessage::Fail() at ??:0
@ 0x7f087b041b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f087b03d7cc google::LogMessage::Flush() at ??:0
@ 0x7f087b03ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f087c4a7044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f087c48bf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f087c489efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f087f03c77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f087f045a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f087f0c676f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f087f0da4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f087f0c5226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f087f0d4c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55574b7a475c std::function<>::operator()() at ??:0
@ 0x7f087c48ee06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f087c48f6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f087c49103b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55574b7a475c std::function<>::operator()() at ??:0
@ 0x7f087c47e807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f087ccbc6db start_thread at ??:0
@ 0x7f087a1f471f clone at ??:0
W20240501 14:10:36.770396 23894 connection.cc:535] client connection to 127.23.83.130:36363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:36.772528 26308 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:36.590502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.83.130:36363 (local address 127.0.0.1:43612)
0501 14:10:36.590906 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:36.590922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:36.591022 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:36.771933 (+180911us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321}
W20240501 14:10:36.773629 26360 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:36.770941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.83.130:36363 (local address 127.0.0.1:43624)
0501 14:10:36.772181 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:36.772195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:36.772284 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:36.773499 (+ 1215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1146,"thread_start_us":68,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | HEALTHY | <none> | 11 | 0
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 1 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 14 | 0
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | HEALTHY | <none> | 10 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.132 | experimental | 127.23.83.132:44579
local_ip_for_outbound_sockets | 127.23.83.134 | experimental | 127.23.83.134:44847
never_fsync | true | unsafe,advanced | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.23.83.132:44579
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.23.83.134:44847
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
time_source | builtin | 127.23.83.134:44847, 127.23.83.129:36771, 127.23.83.131:43707, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.129:36771, tserver@127.23.83.134:44847, and 2 other server(s)
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 61 | Yes
C | A B* C | 3 | 61 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 67 | Yes
B | A B* C | 1 | 67 | Yes
C | [config not available] | | |
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 70 | Yes
B | [config not available] | | |
C | A B C* | 3 | 70 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 61 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 61 | Yes
D | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 67 | Yes
B | [config not available] | | |
C | A B C* | 3 | 67 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 62 | Yes
B | A B* C | 1 | 62 | Yes
C | [config not available] | | |
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 63 | Yes
B | [config not available] | | |
C | A B C* | 3 | 63 | Yes
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 71 | Yes
B | A B* C | 2 | 71 | Yes
C | [config not available] | | |
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | A B C | 1 | 68 | Yes
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 75 | Yes
B | A B* C | 1 | 75 | Yes
C | [config not available] | | |
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 70 | Yes
B | [config not available] | | |
C | A B C* | 2 | 70 | Yes
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 61 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 61 | Yes
D | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is recovering: 1 on-going tablet copies
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): not running [NONVOTER]
State: INITIALIZED
Data state: TABLET_DATA_COPYING
Last status: Tablet Copy: Downloading WAL segment with seq. number 1 (1/1)
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 71 | Yes
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 63 | Yes
B | A B* C | 1 | 63 | Yes
C | [config not available] | | |
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 67 | Yes
C | A* B C | 1 | 67 | Yes
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 67 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 66 | Yes
B | [config not available] | | |
C | A B C* | 1 | 66 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 65 | Yes
B | A B* C | 1 | 65 | Yes
C | [config not available] | | |
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 67 | Yes
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 66 | Yes
B | A B* C | 1 | 66 | Yes
C | [config not available] | | |
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 64 | Yes
B | [config not available] | | |
C | A B C* | 2 | 64 | Yes
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 66 | Yes
B | A B* C | 1 | 66 | Yes
C | [config not available] | | |
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 66 | Yes
B | [config not available] | | |
C | A B C* | 3 | 66 | Yes
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 67 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 68 | Yes
C | A B* C | 1 | 68 | Yes
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 60 | Yes
B | [config not available] | | |
C | A B C* | 2 | 60 | Yes
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 76 | Yes
B | A B* C | 2 | 76 | Yes
C | [config not available] | | |
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 68 | Yes
B | [config not available] | | |
C | A* B C | 3 | 68 | Yes
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 60 | Yes
B | [config not available] | | |
C | A B C* | 1 | 60 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING [LEADER]
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 70 | Yes
B | A B* C | 1 | 70 | Yes
C | [config not available] | | |
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 68 | Yes
C | A B* C | 1 | 68 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B D | 1 | 68 | Yes
C | [config not available] | | |
D | A* B D | 1 | 68 | Yes
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 58 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 58 | Yes
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 65 | Yes
B | [config not available] | | |
C | A B C* | 1 | 65 | Yes
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 65 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 57 | Yes
B | [config not available] | | |
C | A B C* | 3 | 57 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): RUNNING [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 62 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 62 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+-------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 1 | 36 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 14
Median | 21
Third Quartile | 27
Maximum | 40
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
W20240501 14:10:36.908128 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:37.948273 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:38.976577 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:40.032613 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:41.049841 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:42.103544 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:43.172200 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:44.219987 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:45.347259 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:46.414507 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:47.688742 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:48.735021 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:49.998222 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:51.155882 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:52.185585 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:53.251281 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:54.352761 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:55.496958 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:56.672711 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:10:57.674297 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:10:58.914705 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:00.186515 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:01.500355 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:02.848196 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:04.232317 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:05.653443 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:07.111431 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:08.603777 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:09.617882 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:10.913305 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:11.969843 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:13.307071 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:14.393334 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:15.772485 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:16.899070 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:18.325554 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:19.487644 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:20.959185 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:22.153383 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:23.674866 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:24.907092 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:26.469755 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:27.736109 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:29.339038 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
F20240501 14:11:30.460858 29576 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.queue_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.queue_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.run_cpu_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.run_cpu_time_us) (tablet-download-pool-22c34af84b274859a730452846f8e2af.run_wall_time_us, tablet-download-pool-22c34af84b274859a730452846f8e2af.run_wall_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.queue_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.queue_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.run_cpu_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.run_cpu_time_us) (tablet-download-pool-3d969607125e4aa9836046324988984e.run_wall_time_us, tablet-download-pool-3d969607125e4aa9836046324988984e.run_wall_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us) (tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.queue_time_us, tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.queue_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.queue_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.queue_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_cpu_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_cpu_time_us) (tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_wall_time_us, tablet-download-pool-729105456bd0432a8ba5561d35d8adb8.run_wall_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_cpu_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_cpu_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_wall_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.run_wall_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.queue_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.queue_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_cpu_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_cpu_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_wall_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_wall_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_cpu_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_cpu_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_wall_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_wall_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.queue_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.queue_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_cpu_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_cpu_time_us) (tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_wall_time_us, tablet-download-pool-fe34b93d678143f1becc58f0d124396e.run_wall_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.queue_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.queue_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_cpu_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_cpu_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_wall_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fa3ca795dcd google::LogMessage::Fail() at ??:0
@ 0x7fa3ca799b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa3ca7957cc google::LogMessage::Flush() at ??:0
@ 0x7fa3ca796f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa3cbbff044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fa3cbbe3f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa3cbbe1efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fa3ce79477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fa3ce79da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fa3ce81e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fa3ce8324f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fa3ce81d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fa3ce82cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563b6832b75c std::function<>::operator()() at ??:0
@ 0x7fa3cbbe6e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa3cbbe76ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa3cbbe903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563b6832b75c std::function<>::operator()() at ??:0
@ 0x7fa3cbbd6807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3cc4146db start_thread at ??:0
@ 0x7fa3c994c71f clone at ??:0
W20240501 14:11:30.645537 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:32.297346 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:33.303004 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:34.317168 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:35.339509 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:36.370493 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:37.059744 24810 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 3d969607125e4aa9836046324988984e after 343 attempt(s): Failed to write to server: (no server available): Write(tablet: 3d969607125e4aa9836046324988984e, num_ops: 1, num_attempts: 343) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:37.370658 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:38.383896 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:39.391661 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:40.413333 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:41.424993 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:42.431732 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:43.436964 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:44.438153 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:45.452574 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:46.488818 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:47.489744 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:48.510092 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:49.511643 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:11:50.562623 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:51.565654 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:52.606362 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:53.611510 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:54.614674 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:55.620785 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:56.662962 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:57.668365 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:11:58.758193 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:11:59.812003 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:00.829152 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:01.841902 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:02.895618 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:03.900465 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:04.922988 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:05.931703 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:06.945786 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:07.956142 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:08.958614 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:09.976639 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:11.008827 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:12.019145 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:13.040102 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:14.058432 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:15.153005 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:16.169719 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:17.174038 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:18.248482 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:19.265687 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:20.277372 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:21.285542 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:22.434386 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:23.438494 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:24.485168 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:25.497166 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:26.516178 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:27.611843 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:28.660254 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:29.705054 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:30.724282 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:31.927379 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
F20240501 14:12:32.182022 31884 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.queue_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_cpu_time_us) (tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us, tablet-download-pool-030fde7b4e174a96ac6eeb72de037fe3.run_wall_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.queue_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.queue_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_cpu_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_cpu_time_us) (tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_wall_time_us, tablet-download-pool-13724f38c75648b78c4f4385e59e453c.run_wall_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.queue_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_cpu_time_us) (tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us, tablet-download-pool-24a016b48b3349a1b5f684f899fcad04.run_wall_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.queue_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.queue_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_cpu_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_cpu_time_us) (tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_wall_time_us, tablet-download-pool-38e64a0455674ca2b358b487cef8c89f.run_wall_time_us) (tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.queue_time_us, tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.queue_time_us) (tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.run_cpu_time_us, tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.run_cpu_time_us) (tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.run_wall_time_us, tablet-download-pool-44fa9e30b7b140f783568502a245f4ba.run_wall_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.queue_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_cpu_time_us) (tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us, tablet-download-pool-585565d4a38542819e92ee4fb13c7ae3.run_wall_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.queue_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_cpu_time_us) (tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us, tablet-download-pool-6529a0080858471aad76a00c30840708.run_wall_time_us) (tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.queue_time_us, tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.queue_time_us) (tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.run_cpu_time_us, tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.run_cpu_time_us) (tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.run_wall_time_us, tablet-download-pool-6e51873cc1f0480aa4d18488984fc5b8.run_wall_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.queue_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_cpu_time_us) (tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us, tablet-download-pool-85477190db4c440f923d014332113cdc.run_wall_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.queue_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_cpu_time_us) (tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us, tablet-download-pool-8a61c8c45fab43c8a2c2853711d3a668.run_wall_time_us) (tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us, tablet-download-pool-95649d1bf7d249b5856d38b2171674ed.queue_time_us) (tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.queue_time_us, tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.queue_time_us) (tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.run_cpu_time_us, tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.run_cpu_time_us) (tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.run_wall_time_us, tablet-download-pool-abf934e0d6c34788ba0676952c1bc15a.run_wall_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.queue_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.queue_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_cpu_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_cpu_time_us) (tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_wall_time_us, tablet-download-pool-b7e78f49f77042beb5a6bccbe814dd63.run_wall_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.queue_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_cpu_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_cpu_time_us) (tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_wall_time_us, tablet-download-pool-f85b39bf5e85414d8f846d5949c6b25d.run_wall_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.queue_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.queue_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_cpu_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_cpu_time_us) (tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_wall_time_us, tablet-download-pool-fe7ce02f02194b6d9507985bc89dc8e6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fbe23bb8dcd google::LogMessage::Fail() at ??:0
@ 0x7fbe23bbcb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbe23bb87cc google::LogMessage::Flush() at ??:0
@ 0x7fbe23bb9f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe25022044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fbe25006f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fbe25004efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fbe27bb777a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fbe27bc0a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fbe27c4176f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fbe27c554f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fbe27c40226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fbe27c4fc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559387eda75c std::function<>::operator()() at ??:0
@ 0x7fbe25009e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe2500a6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbe2500c03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559387eda75c std::function<>::operator()() at ??:0
@ 0x7fbe24ff9807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe258376db start_thread at ??:0
@ 0x7fbe22d6f71f clone at ??:0
W20240501 14:12:32.425493 23891 connection.cc:535] client connection to 127.23.83.134:44847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:12:32.426311 32008 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:12:32.425863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.83.134:44847 (local address 127.0.0.1:33110)
0501 14:12:32.426154 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 14:12:32.426202 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":205}
W20240501 14:12:32.929289 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:33.931813 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:34.947827 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:35.969458 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:36.998710 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:37.063817 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.129:36771, tserver@127.23.83.131:43707
Tablet Summary
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 177 | Yes
C | A B* C | 3 | 177 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 185 | Yes
B | [config not available] | | |
C | A B C* | 3 | 185 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 182 | Yes
B | [config not available] | | |
C | A B C* | 3 | 182 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 181 | Yes
B | [config not available] | | |
C | A B C* | 1 | 181 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 177 | Yes
B | [config not available] | | |
C | A B C* | 1 | 177 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 3 | 174 | Yes
B | [config not available] | | |
C | A B C* | 3 | 174 | Yes
D | [config not available] | | |
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 31
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 129
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:12:37.999845 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:39.011113 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:40.021698 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:41.023005 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:42.068367 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:43.076844 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:44.146836 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:45.215226 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:46.235841 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:47.320391 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:48.322031 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:49.353101 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:12:50.448132 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:51.457456 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:52.471748 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:53.495474 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:54.578346 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:12:55.700310 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:56.707675 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:12:57.849462 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:58.867915 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:12:59.912107 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:00.976361 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:02.068696 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:03.182551 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:04.324198 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:05.335779 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:06.439970 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:07.453989 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:08.646850 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:09.649500 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:10.673736 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:11.708575 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:12.762807 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:13.835289 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:14.917501 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:16.018034 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:17.131081 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:18.261991 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:19.413197 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:20.573634 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:21.758103 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:22.950829 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:24.163316 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:25.390560 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:26.391104 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:27.569435 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:28.578622 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:29.589718 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:30.778993 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:31.788179 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:32.793217 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:33.813982 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:34.822263 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:35.842054 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:36.846412 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:37.655174 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
W20240501 14:13:37.969259 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:38.970656 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:39.973424 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:40.976560 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:41.985021 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:42.989085 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:44.026990 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:45.061954 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:46.063941 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:47.088376 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:48.140115 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:49.162721 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:50.166568 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:51.196703 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:52.276321 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:53.309085 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:54.377507 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:55.382934 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:13:56.430245 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:13:57.499775 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:13:58.500667 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:13:59.534014 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:00.586052 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:01.660964 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:02.668191 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:03.668419 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:04.675083 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:05.742676 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:06.795011 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:07.802264 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:08.892814 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:09.896169 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:10.914211 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:11.953778 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:13.006181 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:14.072108 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:15.149705 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:16.246487 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:17.265234 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:18.323904 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:19.352120 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:20.354056 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:21.395124 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:22.403688 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:23.474089 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:24.520100 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:25.615085 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:26.677824 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:27.806974 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:28.887885 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:29.928208 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:31.017527 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:32.019081 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:33.110802 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:34.118997 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:35.127122 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:36.129992 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:37.158391 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:37.971483 24808 batcher.cc:441] Timed out: Failed to write batch of 4 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.997s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.131:43707, tserver@127.23.83.129:36771
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:14:38.531013 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:39.534639 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:40.546175 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:14:41.546696 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:42.547174 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:43.553953 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:44.558007 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:45.566499 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:46.568396 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:47.595934 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:48.601099 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:49.613492 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:50.647949 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:51.649715 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:52.650158 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:53.663400 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:14:54.671308 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:55.677418 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:56.704625 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:57.705691 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:14:58.730526 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:14:59.748632 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:00.761812 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:01.777112 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:02.866746 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:03.868948 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:04.900256 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:05.906801 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:06.931577 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:07.967921 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:08.973600 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:09.977834 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:10.986068 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:11.987378 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:13.021019 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:14.076751 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:15.152155 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:16.242914 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:17.277977 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:18.282997 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:19.322923 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:20.405656 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:21.409193 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:22.527268 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:23.555964 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:24.624243 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:25.630550 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:26.685691 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:27.707075 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:28.711580 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:29.745887 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:15:30.807128 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:31.869482 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:32.876579 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:33.908035 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:34.913101 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:35.928025 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:36.947296 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:37.974375 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:38.529294 24808 batcher.cc:441] Timed out: Failed to write batch of 4 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.995s (SENT)
W20240501 14:15:38.974496 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:39.975153 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:15:40.989388 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:15:41.990928 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:43.006423 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:44.031200 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:45.033712 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:46.035933 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:47.051781 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:48.075109 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:49.078009 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:50.088985 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:51.097342 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:52.104094 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:53.108253 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:54.133404 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:55.134666 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:15:56.151623 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:15:57.154603 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:15:58.216183 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:15:59.233709 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:00.236294 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:01.276937 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:02.351258 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:03.447993 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:04.458735 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:05.479679 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:06.510828 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:07.594116 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:08.596251 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:09.602337 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:10.603981 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:11.617261 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:12.630957 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:13.651746 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:14.718839 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:15.799103 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:16.893302 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:18.005905 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:19.136543 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:20.284320 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:21.441633 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:22.442086 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:16:23.504312 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:24.514864 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:25.617781 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:26.636381 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:27.783440 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:28.786978 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:29.793028 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:30.795445 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:31.938644 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:32.957767 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:33.967967 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:34.968302 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:35.976550 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:36.980060 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:38.008126 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:38.870510 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.995s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.129:36771, tserver@127.23.83.131:43707
Tablet Summary
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:16:39.407227 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:40.409027 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:41.414884 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:16:42.419701 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:43.424038 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:44.434597 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:45.438659 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:46.449707 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:47.452718 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:48.481969 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:49.493309 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:16:50.494704 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:51.532975 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:52.537833 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:53.541975 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:54.555696 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:55.560300 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:56.609055 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:16:57.616895 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:16:58.622818 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:16:59.632100 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:00.636709 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:01.639940 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:02.691851 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:03.785744 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:04.828109 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:05.832329 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:06.974201 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:07.976212 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:09.059993 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:10.117079 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:11.117439 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:12.135099 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:13.164610 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:14.210651 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:15.277922 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:16.359189 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:17.457101 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:18.458226 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:19.479619 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:20.556496 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:21.714395 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:22.764878 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:23.770743 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:24.849988 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:25.877406 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:26.877521 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:27.913208 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:28.976591 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:30.006088 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:31.086166 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:32.176163 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:33.194759 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:34.206286 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:35.210839 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:36.216265 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:37.228137 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:38.231288 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:39.284700 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:39.408267 24808 batcher.cc:441] Timed out: Failed to write batch of 2 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
W20240501 14:17:40.289831 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:41.307221 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:42.313808 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:43.317976 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:44.338409 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:45.356813 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:46.407935 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:47.419211 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:48.447265 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:49.467435 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:17:50.469864 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:51.520727 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:52.533846 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:53.600361 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:54.620661 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:55.674157 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:17:56.675274 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:17:57.702896 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:58.703588 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:17:59.748533 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:00.765024 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:01.804718 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:02.820072 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:03.847121 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:04.849633 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:05.865757 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:06.912003 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:07.965256 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:08.967245 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:09.969206 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:11.018828 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:12.033603 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:13.053992 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:14.100147 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:15.157096 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:16.230899 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:17.232117 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:18.269604 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:19.392823 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:20.405166 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:21.405767 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:22.409067 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:23.438395 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:24.476871 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:25.531662 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:26.532687 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:27.536026 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:28.604389 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:29.621075 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:30.656208 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:31.783200 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:32.786593 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:33.816269 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:34.826658 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:35.827386 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:36.834910 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:37.837666 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:38.865978 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:39.695065 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.994s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.131:43707, tserver@127.23.83.129:36771
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:18:40.263355 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:41.274933 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:42.303714 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:43.305168 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:44.310631 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:45.342264 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:46.342398 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:47.353200 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:48.354193 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:49.380931 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:50.389544 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:18:51.407584 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:52.430749 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:18:53.431187 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:54.496662 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:55.526139 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:56.597797 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:57.616340 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:18:58.634219 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:18:59.667279 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:00.676887 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:01.685000 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:02.692231 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:03.741833 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:04.750671 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:05.751156 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:06.757153 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:07.800119 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:08.841954 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:09.852113 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:10.931728 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:11.933218 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:12.949795 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:13.980046 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:15.028237 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:16.091063 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:17.174206 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:18.273515 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:19.386833 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:20.513712 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:21.523555 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:22.526129 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:23.556643 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:24.582803 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:25.584448 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:26.690202 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:27.718303 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:28.847112 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:29.895908 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:30.909215 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:32.015091 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:33.099514 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:34.109417 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:35.123222 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:36.134222 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:37.141645 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:38.157454 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:39.174523 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:40.177562 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:40.263667 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.992s (SENT)
W20240501 14:19:41.177888 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:42.185036 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:43.190747 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:44.195137 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:45.197748 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:46.202502 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:47.203073 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:48.229140 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:49.250916 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:50.251075 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:19:51.251744 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:52.255918 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:19:53.259187 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:54.262907 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:55.273641 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:56.283723 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:19:57.290562 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:58.293267 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:19:59.335450 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:00.352469 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:01.362299 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:02.385934 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:03.420270 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:04.503286 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:05.532965 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:06.533699 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:07.552834 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:08.612697 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:09.623692 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:10.661651 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:11.676275 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:12.684269 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:13.802881 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:14.841655 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:15.898484 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:16.970901 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:18.056627 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:19.065865 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:20.067767 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:21.126333 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:22.127245 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:23.142522 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:24.217617 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:25.228156 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:26.416025 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:27.483887 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:28.560357 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:29.577087 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:30.599422 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:31.708124 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:32.757716 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:33.970155 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:34.993741 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:36.039341 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:37.047628 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:38.051813 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:39.077378 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:40.101336 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:40.574072 24808 batcher.cc:441] Timed out: Failed to write batch of 2 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.131:43707, tserver@127.23.83.129:36771
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:20:41.137856 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:42.140571 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:43.155629 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:44.159852 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:45.162649 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:46.201640 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:47.201917 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:48.202903 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:20:49.209189 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:50.214236 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:51.221081 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:52.242626 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:53.266487 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:54.271643 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:20:55.288064 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:56.318262 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:57.341753 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:20:58.379511 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:20:59.392041 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:00.398358 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:01.485908 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:02.499403 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:03.540508 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:04.613224 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:05.616163 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:06.623795 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:07.642665 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:08.680735 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:09.699437 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:10.758342 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:11.814682 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:12.815579 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:13.831897 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:14.867111 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:15.922657 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:16.982707 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:18.062700 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:19.160073 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:20.172269 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:21.207810 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:22.260087 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:23.260939 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:24.288956 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:25.296730 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:26.367216 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:27.373191 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:28.379909 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:29.421566 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:30.464252 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:31.474375 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:32.576280 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:33.597923 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:34.628732 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:35.649444 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:36.663264 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:37.673274 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:38.691113 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:39.706633 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:40.709419 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:41.138300 24808 batcher.cc:441] Timed out: Failed to write batch of 5 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.997s (SENT)
W20240501 14:21:41.710310 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:42.710948 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:43.711494 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:44.718186 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:45.725540 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:46.732192 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:47.752682 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:48.756089 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:49.761265 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:50.781359 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:51.814138 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:52.841622 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:21:53.858520 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:54.890348 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:55.897152 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:21:56.931177 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:21:57.937944 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:58.987637 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:21:59.998178 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:01.006888 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:02.037405 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:03.064327 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:04.066733 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:05.112511 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:06.157873 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:07.158831 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:08.189425 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:09.191134 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:10.193542 24810 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:11.198251 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:12.210544 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:13.211236 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:14.212544 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:15.226748 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:16.228020 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:17.252187 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:18.312578 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:19.321753 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:20.348847 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:21.374639 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:22.409592 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:23.410687 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:24.438961 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:25.452574 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:26.467602 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:27.528930 24810 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:28.551389 24810 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:29.567142 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:30.602435 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:31.609041 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:32.626538 24807 meta_cache.cc:302] tablet 9a8785261879441d998a669efd4f06f3: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:33.629930 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:34.655226 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:35.666944 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:36.739421 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:37.742506 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:38.753108 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:39.773699 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:40.806452 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:22:41.440578 24808 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 6529a0080858471aad76a00c30840708 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.129:36771, tserver@127.23.83.131:43707
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:22:42.004964 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:43.010696 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:44.018162 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:45.031301 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:46.036747 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:47.064530 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:48.067255 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:49.071627 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:50.082624 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:51.097447 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:52.102108 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:22:53.103371 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:54.135499 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:55.147886 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:56.246296 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:57.284338 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:22:58.356475 24810 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:22:59.456348 24807 meta_cache.cc:302] tablet 91cbe050b9854309b6c893942c48affd: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:00.463472 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:01.572427 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:02.572732 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:03.599761 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:04.650887 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:05.731323 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:06.831852 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:07.963887 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:08.975111 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:10.057404 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:11.078418 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:12.229477 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:13.234493 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:14.240391 24810 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:15.248899 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:16.288051 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:17.340175 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:18.408854 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:19.488658 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:20.592784 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:21.711051 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:22.845523 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:23.997794 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:24.999006 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:26.056730 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:27.079602 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:28.159963 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:29.160008 24807 meta_cache.cc:302] tablet 0c82567aea79452a9668bc5172abad1f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:30.303726 24810 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:31.322134 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:32.323433 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:33.459306 24807 meta_cache.cc:302] tablet 1ce14c9f93814647b255fae7853cf8bf: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:34.479832 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:35.487563 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:36.510107 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:37.525969 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:38.529934 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:39.548203 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:40.569425 24807 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:41.573657 24807 meta_cache.cc:302] tablet 9c81b20f4bcd4ea6b078d40caa84e885: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:42.003448 24808 batcher.cc:441] Timed out: Failed to write batch of 3 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
W20240501 14:23:42.578375 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:43.593686 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:44.594692 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:45.602888 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:46.605280 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:47.618580 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:48.644423 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:49.696363 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:50.710714 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:51.712692 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:52.772238 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:53.807390 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:23:54.823261 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:55.870611 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:56.876025 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:23:57.913759 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:23:58.923871 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:23:59.943559 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:00.950748 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:01.972540 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:02.974694 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:04.009917 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:05.022661 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:06.033943 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:07.075033 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:08.090162 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:09.122664 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:10.135355 24810 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:11.199662 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:12.219174 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:13.242791 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:14.247634 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:15.265550 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:16.306080 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:17.310334 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:18.322930 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:19.328696 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:20.371594 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:21.383780 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:22.384058 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:23.418541 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:24.466372 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:25.469579 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:26.527577 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:27.530100 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:28.551417 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:29.608733 24810 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:30.612551 24810 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:31.640204 24810 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:32.672888 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:33.740306 24810 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
W20240501 14:24:34.771068 24807 meta_cache.cc:302] tablet a915c02a73cf448091632815ff49557f: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:35.801309 24807 meta_cache.cc:302] tablet 12cab77f880d490b8a72d56a0629efe8: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:36.883795 24810 meta_cache.cc:302] tablet 729105456bd0432a8ba5561d35d8adb8: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:37.944237 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:39.007293 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:40.017731 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:41.044374 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:42.074216 24807 meta_cache.cc:302] tablet dd4bf00a3af5428ba5fc5a70eedaa348: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:42.299162 24808 batcher.cc:441] Timed out: Failed to write batch of 2 ops to tablet 030fde7b4e174a96ac6eeb72de037fe3 after 1 attempt(s): Failed to write to server: c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): Write RPC to 127.23.83.131:43707 timed out after 59.998s (SENT)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e4494bc0912542b4a71b00a10d036dd7 | 127.23.83.190:33869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.23.83.148:41783 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
86e43abaefd642c3ba7d98bd3bae8b21 | 127.23.83.129:36771 | HEALTHY | <none> | 10 | 0
c8e924dcdb9d43c7916f608ebbf71dcb | 127.23.83.131:43707 | HEALTHY | <none> | 17 | 0
1ebfab2167974c67908ba74179b736be | 127.23.83.130:36363 | UNAVAILABLE | <none> | n/a | n/a
601fcf5e368d40288a9db33f7b4e060f | 127.23.83.134:44847 | UNAVAILABLE | <none> | n/a | n/a
77a65181cc774449b32220b150f27ea1 | 127.23.83.133:45453 | UNAVAILABLE | <none> | n/a | n/a
f3cbf2729d8a45c3bbd222e800d2fffb | 127.23.83.132:44579 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.23.83.130:36363: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.134:44847: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.133:45453: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.23.83.132:44579: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------------------------
ipki_server_key_size | 768 | experimental | 127.23.83.129:36771, 127.23.83.131:43707
local_ip_for_outbound_sockets | 127.23.83.129 | experimental | 127.23.83.129:36771
local_ip_for_outbound_sockets | 127.23.83.131 | experimental | 127.23.83.131:43707
never_fsync | true | unsafe,advanced | 127.23.83.129:36771, 127.23.83.131:43707
openssl_security_level_override | 0 | unsafe,hidden | 127.23.83.129:36771, 127.23.83.131:43707
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.23.83.129:36771, 127.23.83.131:43707
rpc_server_allow_ephemeral_ports | true | unsafe | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_format | pb | hidden | 127.23.83.129:36771, 127.23.83.131:43707
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.23.83.129:36771
server_dump_info_path | /tmp/dist-test-taskWs8th6/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572617849825-23886-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.23.83.131:43707
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.23.83.129:36771, 127.23.83.131:43707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+------------------------------------------
builtin_ntp_servers | 127.23.83.148:41783 | 127.23.83.129:36771, 127.23.83.131:43707
time_source | builtin | 127.23.83.129:36771, 127.23.83.131:43707
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.23.83.190:33869, tserver@127.23.83.129:36771, tserver@127.23.83.131:43707
Tablet Summary
Tablet f85b39bf5e85414d8f846d5949c6b25d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 229 | Yes
C | A B* C | 3 | 229 | Yes
Tablet 33edb7a66a7a4f2c8f397a521da5fda2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 6e51873cc1f0480aa4d18488984fc5b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 238 | Yes
B | [config not available] | | |
C | A B C* | 3 | 238 | Yes
Tablet f44967f8352f422189d3e5b4a8ddbc0f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 38e64a0455674ca2b358b487cef8c89f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 235 | Yes
B | [config not available] | | |
C | A B C* | 3 | 235 | Yes
Tablet 4248d5ae85e34b969fa59c0320c8eae4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 76 | Yes
C | A B C* | 2 | 76 | Yes
Tablet b7e78f49f77042beb5a6bccbe814dd63 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 121 | Yes
B | A B* C | 3 | 121 | Yes
C | [config not available] | | |
Tablet 06f46b6599104821b6446e5c5d94d2d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 3d969607125e4aa9836046324988984e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 068eeaab02144dec9e09f85852476dc2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 89 | Yes
C | A B C* | 2 | 89 | Yes
Tablet 24a016b48b3349a1b5f684f899fcad04 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 182 | Yes
B | A B* C | 2 | 182 | Yes
C | [config not available] | | |
Tablet 1ce14c9f93814647b255fae7853cf8bf of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 22c34af84b274859a730452846f8e2af of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 14dad87a8a604796afb5f6ca72adee32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet 6529a0080858471aad76a00c30840708 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | No
C | [config not available] | | |
Tablet bbff0f0cb864400f9268f92a6c6cca6f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 95649d1bf7d249b5856d38b2171674ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 234 | Yes
B | [config not available] | | |
C | A B C* | 1 | 234 | Yes
Tablet 402e8cd189774227941101fd68782a89 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet fe34b93d678143f1becc58f0d124396e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 77a65181cc774449b32220b150f27ea1
D = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d0a640c883a14d1e8715a534595a6877 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 585565d4a38542819e92ee4fb13c7ae3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 179 | Yes
B | A B* C | 2 | 179 | Yes
C | [config not available] | | |
Tablet 7d42976bfa624cdd82cf06b6e2e41c8d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet fe7ce02f02194b6d9507985bc89dc8e6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 180 | Yes
B | A B* C | 3 | 180 | Yes
C | [config not available] | | |
Tablet a915c02a73cf448091632815ff49557f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 030fde7b4e174a96ac6eeb72de037fe3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable [NONVOTER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
D = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D | 1 | 120 | No
C | [config not available] | | |
D | [config not available] | | |
Tablet 9c81b20f4bcd4ea6b078d40caa84e885 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet abf934e0d6c34788ba0676952c1bc15a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 174 | Yes
B | A B* C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2eccecd65b8c4ad49eb3414afd96ab90 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 44fa9e30b7b140f783568502a245f4ba of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 181 | Yes
B | A* B C | 3 | 181 | Yes
C | [config not available] | | |
Tablet 12cab77f880d490b8a72d56a0629efe8 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 8a61c8c45fab43c8a2c2853711d3a668 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | 230 | Yes
B | [config not available] | | |
C | A B C* | 1 | 230 | Yes
Tablet 7153494f545a4a12a9650e1f5f58c1ed of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 84 | Yes
C | A B C* | 2 | 84 | Yes
Tablet 85477190db4c440f923d014332113cdc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
All reported replicas are:
A = c8e924dcdb9d43c7916f608ebbf71dcb
B = 601fcf5e368d40288a9db33f7b4e060f
C = 86e43abaefd642c3ba7d98bd3bae8b21
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 126 | Yes
B | [config not available] | | |
C | A* B C | 1 | 126 | Yes
Tablet 0c82567aea79452a9668bc5172abad1f of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 729105456bd0432a8ba5561d35d8adb8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
All reported replicas are:
A = 1ebfab2167974c67908ba74179b736be
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = f3cbf2729d8a45c3bbd222e800d2fffb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | [config not available] | | |
Tablet dd4bf00a3af5428ba5fc5a70eedaa348 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable [NONVOTER]
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
D = 1ebfab2167974c67908ba74179b736be
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 13724f38c75648b78c4f4385e59e453c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = c8e924dcdb9d43c7916f608ebbf71dcb
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 179 | Yes
B | A B* C | 1 | 179 | Yes
C | [config not available] | | |
Tablet 91cbe050b9854309b6c893942c48affd of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet 183584db7f7a4468aa534a0a334280ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING
1ebfab2167974c67908ba74179b736be (127.23.83.130:36363): TS unavailable
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING [LEADER]
All reported replicas are:
A = 86e43abaefd642c3ba7d98bd3bae8b21
B = 1ebfab2167974c67908ba74179b736be
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 226 | Yes
B | [config not available] | | |
C | A B C* | 3 | 226 | Yes
Tablet 9a8785261879441d998a669efd4f06f3 of table 'RemoveReplaceInCycle' is unavailable: 3 replica(s) not RUNNING
f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579): TS unavailable [LEADER]
77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453): TS unavailable
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
All reported replicas are:
A = f3cbf2729d8a45c3bbd222e800d2fffb
B = 77a65181cc774449b32220b150f27ea1
C = 601fcf5e368d40288a9db33f7b4e060f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | [config not available] | | |
Tablet bbdd248d62584dd89a8eb6f7664ac5ca of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847): TS unavailable
86e43abaefd642c3ba7d98bd3bae8b21 (127.23.83.129:36771): RUNNING [LEADER]
c8e924dcdb9d43c7916f608ebbf71dcb (127.23.83.131:43707): RUNNING
All reported replicas are:
A = 601fcf5e368d40288a9db33f7b4e060f
B = 86e43abaefd642c3ba7d98bd3bae8b21
C = c8e924dcdb9d43c7916f608ebbf71dcb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 7 | Yes
C | A B* C | 2 | 7 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 0 | 0 | 24 | 16
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 25
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 4 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 4 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 14:24:43.075026 24807 meta_cache.cc:302] tablet 3d969607125e4aa9836046324988984e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
W20240501 14:24:44.094831 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:45.103986 24810 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 1ebfab2167974c67908ba74179b736be (127.23.83.130:36363) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.130:36363: connect: Connection refused (error 111)
W20240501 14:24:46.105209 24807 meta_cache.cc:302] tablet bbff0f0cb864400f9268f92a6c6cca6f: replica 601fcf5e368d40288a9db33f7b4e060f (127.23.83.134:44847) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.134:44847: connect: Connection refused (error 111)
W20240501 14:24:47.116140 24807 meta_cache.cc:302] tablet 22c34af84b274859a730452846f8e2af: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
************************ BEGIN STACKS **************************
W20240501 14:24:48.118829 24810 meta_cache.cc:302] tablet f44967f8352f422189d3e5b4a8ddbc0f: replica f3cbf2729d8a45c3bbd222e800d2fffb (127.23.83.132:44579) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.132:44579: connect: Connection refused (error 111)
[New LWP 23889]
[New LWP 23890]
[New LWP 23891]
[New LWP 23892]
[New LWP 23893]
[New LWP 23894]
[New LWP 23895]
[New LWP 24776]
[New LWP 24777]
[New LWP 24778]
[New LWP 24779]
[New LWP 24807]
[New LWP 24808]
[New LWP 24809]
[New LWP 24810]
[New LWP 25292]
[New LWP 25293]
[New LWP 4073]
[New LWP 4158]
[New LWP 4161]
[New LWP 4162]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7fdf09302380 (LWP 23886) "raft_consensus_" 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7fdf072dd700 (LWP 23889) "raft_consensus_" 0x00007fdf159bc32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7fdf05900700 (LWP 23890) "kernel-watcher-" 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7fdf050ff700 (LWP 23891) "rpc reactor-238" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7fdf048fe700 (LWP 23892) "rpc reactor-238" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7fdf040fd700 (LWP 23893) "rpc reactor-238" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7fdf038fc700 (LWP 23894) "rpc reactor-238" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7fdf030fb700 (LWP 23895) "rpc reactor-238" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7fdf028fa700 (LWP 24776) "rpc reactor-247" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7fdf01ef2700 (LWP 24777) "rpc reactor-247" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7fdf016f1700 (LWP 24778) "rpc reactor-247" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7fdf00ef0700 (LWP 24779) "rpc reactor-247" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7fdeff6ed700 (LWP 24807) "rpc reactor-248" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7fdefeeec700 (LWP 24808) "rpc reactor-248" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7fdefe6eb700 (LWP 24809) "rpc reactor-248" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7fdefdeea700 (LWP 24810) "rpc reactor-248" 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7fdefd6e9700 (LWP 25292) "raft_consensus_" 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
18 Thread 0x7fdf006ef700 (LWP 25293) "raft_consensus_" 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
19 Thread 0x7fdefc6e7700 (LWP 4073) "raft_consensus_" 0x00007fdf159b7ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
20 Thread 0x7fdefaee4700 (LWP 4158) "client-negotiat" 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
21 Thread 0x7fdeffeee700 (LWP 4161) "client-negotiat" 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
22 Thread 0x7fdefcee8700 (LWP 4162) "client-negotiat" 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 22 (Thread 0x7fdefcee8700 (LWP 4162)):
#0 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf154769d4 in kudu::ConditionVariable::WaitFor (this=0x7fdefcee7690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007fdf155c48d1 in kudu::ThreadPool::DispatchThread (this=0x5619833acf00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:720
#3 0x00007fdf155c56ef in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x561983405f58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:848
#4 0x00007fdf155c703b in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#5 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x561983405f58) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x561983405f00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 21 (Thread 0x7fdeffeee700 (LWP 4161)):
#0 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf154769d4 in kudu::ConditionVariable::WaitFor (this=0x7fdeffeed690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007fdf155c48d1 in kudu::ThreadPool::DispatchThread (this=0x56198325ef00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:720
#3 0x00007fdf155c56ef in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x561983768c58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:848
#4 0x00007fdf155c703b in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#5 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x561983768c58) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x561983768c00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 20 (Thread 0x7fdefaee4700 (LWP 4158)):
#0 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf154769d4 in kudu::ConditionVariable::WaitFor (this=0x7fdefaee3690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007fdf155c48d1 in kudu::ThreadPool::DispatchThread (this=0x5619833acf00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:720
#3 0x00007fdf155c56ef in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x561983494e58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:848
#4 0x00007fdf155c703b in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#5 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x561983494e58) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x561983494e00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 19 (Thread 0x7fdefc6e7700 (LWP 4073)):
#0 0x00007fdf159b7ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf15476665 in kudu::ConditionVariable::Wait (this=0x561983496e10) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:55
#2 0x00007fdf162004c1 in kudu::client::KuduSession::Data::Flush (this=0x561983496d80) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/session-internal.cc:257
#3 0x00007fdf1610a638 in kudu::client::KuduSession::Flush (this=0x561983357a40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client.cc:1524
#4 0x00007fdf16a317e3 in kudu::TestWorkload::WriteThread (this=0x7fffcb6ab9d0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/test_workload.cc:204
#5 0x00007fdf16a3bb8f in std::__invoke_impl<void, void (kudu::TestWorkload::*)(), kudu::TestWorkload*> (__f=@0x56198363fa50: (void (kudu::TestWorkload::*)(kudu::TestWorkload * const)) 0x7fdf16a30d88 <kudu::TestWorkload::WriteThread()>, __t=@0x56198363fa48: 0x7fffcb6ab9d0) at /usr/include/c++/7/bits/invoke.h:73
#6 0x00007fdf16a3b436 in std::__invoke<void (kudu::TestWorkload::*)(), kudu::TestWorkload*> (__fn=@0x56198363fa50: (void (kudu::TestWorkload::*)(kudu::TestWorkload * const)) 0x7fdf16a30d88 <kudu::TestWorkload::WriteThread()>) at /usr/include/c++/7/bits/invoke.h:95
#7 0x00007fdf16a3cb07 in std::thread::_Invoker<std::tuple<void (kudu::TestWorkload::*)(), kudu::TestWorkload*> >::_M_invoke<0ul, 1ul> (this=0x56198363fa48) at /usr/include/c++/7/thread:234
#8 0x00007fdf16a3ca6d in std::thread::_Invoker<std::tuple<void (kudu::TestWorkload::*)(), kudu::TestWorkload*> >::operator() (this=0x56198363fa48) at /usr/include/c++/7/thread:243
#9 0x00007fdf16a3c9e2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (kudu::TestWorkload::*)(), kudu::TestWorkload*> > >::_M_run (this=0x56198363fa40) at /usr/include/c++/7/thread:186
#10 0x00007fdf1457f6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 18 (Thread 0x7fdf006ef700 (LWP 25293)):
#0 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf14b3e282 in base::SleepForNanoseconds (nanoseconds=1000000000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/gutil/sysinfo.cc:96
#2 0x00007fdf15554a69 in kudu::SleepFor (delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/monotime.cc:264
#3 0x0000561980e2d9cc in kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()>::operator()(void) const (__closure=0x5619833cf7a8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:215
#4 0x0000561980e30299 in std::__invoke_impl<void, kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> >(std::__invoke_other, kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#5 0x0000561980e2fef9 in std::__invoke<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> >(kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#6 0x0000561980e30d36 in std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x5619833cf7a8) at /usr/include/c++/7/thread:234
#7 0x0000561980e30ce3 in std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > >::operator()(void) (this=0x5619833cf7a8) at /usr/include/c++/7/thread:243
#8 0x0000561980e30ca2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > > >::_M_run(void) (this=0x5619833cf7a0) at /usr/include/c++/7/thread:186
#9 0x00007fdf1457f6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 17 (Thread 0x7fdefd6e9700 (LWP 25292)):
#0 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf14b3e282 in base::SleepForNanoseconds (nanoseconds=250000000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/gutil/sysinfo.cc:96
#2 0x00007fdf15554a69 in kudu::SleepFor (delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/monotime.cc:264
#3 0x0000561980e2d52c in kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()>::operator()(void) const (__closure=0x5619833cf5f8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:196
#4 0x0000561980e3012f in std::__invoke_impl<void, kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> >(std::__invoke_other, kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#5 0x0000561980e2fdf6 in std::__invoke<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> >(kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#6 0x0000561980e30d62 in std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x5619833cf5f8) at /usr/include/c++/7/thread:234
#7 0x0000561980e30d07 in std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > >::operator()(void) (this=0x5619833cf5f8) at /usr/include/c++/7/thread:243
#8 0x0000561980e30cc2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody()::<lambda()> > > >::_M_run(void) (this=0x5619833cf5f0) at /usr/include/c++/7/thread:186
#9 0x00007fdf1457f6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7fdefdeea700 (LWP 24810)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983352880, timeout=0.073888948209969385) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983352880, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b3560, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b3558) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x561983448658) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x561983448658) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x561983448600) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7fdefe6eb700 (LWP 24809)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983352400, timeout=0.099351610209396313) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983352400, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b33a0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b3398) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x561983448558) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x561983448558) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x561983448500) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7fdefeeec700 (LWP 24808)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983351f80, timeout=0.086823947000084445) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983351f80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b31e0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b31d8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bee58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bee58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bee00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7fdeff6ed700 (LWP 24807)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x5619833c0d80, timeout=0.069478967000122793) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x5619833c0d80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b3020, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b3018) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bef58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bef58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bef00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7fdf00ef0700 (LWP 24779)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983351b00, timeout=0.099260336209681554) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983351b00, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2e60, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2e58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831be358) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831be358) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831be300) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7fdf016f1700 (LWP 24778)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983351680, timeout=0.099496679209551075) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983351680, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2ca0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2c98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831be958) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831be958) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831be900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7fdf01ef2700 (LWP 24777)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x5619833c1680, timeout=0.099263219209660747) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x5619833c1680, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2ae0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2ad8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831be558) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831be558) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831be500) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7fdf028fa700 (LWP 24776)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x5619833c1200, timeout=0.099802281209576904) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x5619833c1200, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2920, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2918) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831be758) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831be758) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831be700) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7fdf030fb700 (LWP 23895)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983350d80, timeout=0.099128140208449622) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983350d80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2220, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2218) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bf458) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bf458) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bf400) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7fdf038fc700 (LWP 23894)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983350900, timeout=0.099931254208058817) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983350900, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b23e0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b23d8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bf058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bf058) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bf000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7fdf040fd700 (LWP 23893)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983350480, timeout=0.099759035208535352) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983350480, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2760, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2758) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831be858) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831be858) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831be800) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7fdf048fe700 (LWP 23892)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983350000, timeout=0.099986051208361459) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983350000, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b2060, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bf258) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bf258) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bf200) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7fdf050ff700 (LWP 23891)):
#0 0x00007fdf13c3ca47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fdf0f43fde0 in epoll_poll (loop=0x561983213600, timeout=0.099450679208075599) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fdf0f44349e in ev_run (loop=0x561983213600, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fdf155adba3 in ev::loop_ref::run (this=0x5619832b25a0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fdf11dc231d in kudu::rpc::ReactorThread::RunThread (this=0x5619832b2598) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5 0x00007fdf11dbec49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5619831bec58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6 0x00007fdf11dc6934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bec58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bec00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7fdf05900700 (LWP 23890)):
#0 0x00007fdf159b7fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf154767eb in kudu::ConditionVariable::WaitUntil (this=0x5619831a72d8, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:87
#2 0x00007fdf1654efde in kudu::CountDownLatch::WaitUntil (this=0x5619831a72a0, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:87
#3 0x00007fdf1654f088 in kudu::CountDownLatch::WaitFor (this=0x5619831a72a0, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:97
#4 0x00007fdf1550e166 in kudu::KernelStackWatchdog::RunThread (this=0x5619831a7200) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007fdf1550d375 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x5619831bf158) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007fdf1550eaca in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00007fdf16841856 in std::function<void ()>::operator()() const (this=0x5619831bf158) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007fdf155b4807 in kudu::Thread::SuperviseThread (arg=0x5619831bf100) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7fdf072dd700 (LWP 23889)):
#0 0x00007fdf159bc32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf155ab7b5 in kudu::Subprocess::DoWait (this=0x7fdf072dc380, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:817
#2 0x00007fdf155a8be8 in kudu::Subprocess::Wait (this=0x7fdf072dc380, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:544
#3 0x00007fdf154fbb7c in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:258
#4 0x00007fdf154fb6ab in kudu::PstackWatcher::RunGdbStackDump (pid=23886, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:218
#5 0x00007fdf154faf09 in kudu::PstackWatcher::DumpPidStacks (pid=23886, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:169
#6 0x00007fdf154fae82 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:162
#7 0x00007fdf168ec087 in kudu::<lambda()>::operator()(void) const (__closure=0x5619832259f8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:60
#8 0x00007fdf168ec888 in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9 0x00007fdf168ec5dc in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x00007fdf168ece6e in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x5619832259f8) at /usr/include/c++/7/thread:234
#11 0x00007fdf168ece13 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x5619832259f8) at /usr/include/c++/7/thread:243
#12 0x00007fdf168ecdce in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x5619832259f0) at /usr/include/c++/7/thread:186
#13 0x00007fdf1457f6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fdf159b16db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007fdf13c3c71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7fdf09302380 (LWP 23886)):
#0 0x00007fdf159bbd50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fdf14b3e282 in base::SleepForNanoseconds (nanoseconds=1000000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/gutil/sysinfo.cc:96
#2 0x00007fdf15554a69 in kudu::SleepFor (delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/monotime.cc:264
#3 0x0000561980e2e8e1 in kudu::tserver::RaftConsensusStressITest_RemoveReplaceInCycle_Test::TestBody (this=0x561983213a80) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:229
#4 0x00007fdf157570ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fdf1575882b "the test body", method=<optimized out>, object=0x561983213a80) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#5 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x561983213a80, method=<optimized out>, location=location@entry=0x7fdf1575882b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#6 0x00007fdf1574bbdb in testing::Test::Run (this=0x561983213a80) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#7 0x00007fdf1574bd9d in testing::TestInfo::Run (this=0x5619831a6240) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#8 0x00007fdf1574c377 in testing::TestSuite::Run (this=0x5619831a6480) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#9 0x00007fdf1574c77c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#10 0x00007fdf1575760d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fdf1575a3d8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x56198325e280) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#11 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x56198325e280, method=<optimized out>, location=location@entry=0x7fdf1575a3d8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#12 0x00007fdf1574be63 in testing::UnitTest::Run (this=0x7fdf1596b1e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#13 0x00007fdf168ed1de in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#14 0x00007fdf168ec53d in main (argc=1, argv=0x7fffcb6ac058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:109
#15 0x00007fdf13b3cbf7 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#16 0x0000561980e2ce4a in _start ()
W20240501 14:25:19.629738 24807 meta_cache.cc:302] tablet fe34b93d678143f1becc58f0d124396e: replica 77a65181cc774449b32220b150f27ea1 (127.23.83.133:45453) has failed: Network error: Client connection negotiation failed: client connection to 127.23.83.133:45453: connect: Connection refused (error 111)
************************* END STACKS ***************************
F20240501 14:25:19.701196 23889 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1714573519 (unix time) try "date -d @1714573519" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005d4e) received by PID 23886 (TID 0x7fdf072dd700) from PID 23886; stack trace: ***
@ 0x7fdf159bc980 (unknown) at ??:0
@ 0x7fdf13b59fb7 gsignal at ??:0
@ 0x7fdf13b5b921 abort at ??:0
@ 0x7fdf1485edcd google::LogMessage::Fail() at ??:0
@ 0x7fdf14862b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fdf1485e7cc google::LogMessage::Flush() at ??:0
@ 0x7fdf1485ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdf168ec1c4 _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at ??:0
@ 0x7fdf168ec888 _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x7fdf168ec5dc _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at ??:0
@ 0x7fdf168ece6e _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x7fdf168ece13 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at ??:0
@ 0x7fdf168ecdce _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at ??:0
@ 0x7fdf1457f6df (unknown) at ??:0
@ 0x7fdf159b16db start_thread at ??:0
@ 0x7fdf13c3c71f clone at ??:0