[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:56:42.118208 19765 test_util.cc:268] Using random seed: -492117968
I20240501 19:56:42.124423 19765 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:42.124475 19765 ts_itest-base.cc:113] --------------
I20240501 19:56:42.124485 19765 ts_itest-base.cc:114] 6 tablet servers
I20240501 19:56:42.124495 19765 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:42.124504 19765 ts_itest-base.cc:116] --------------
2024-05-01T19:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:42Z Disabled control of system clock
I20240501 19:56:42.133368 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.77.126:38343
--webserver_interface=127.19.77.126
--webserver_port=0
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.77.126:38343
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=40 with env {}
I20240501 19:56:42.297919 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 19777
I20240501 19:56:42.298058 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:42.353901 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.65:0
--local_ip_for_outbound_sockets=127.19.77.65
--webserver_interface=127.19.77.65
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:42.508826 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 19868
I20240501 19:56:42.508929 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:42.510826 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.66:0
--local_ip_for_outbound_sockets=127.19.77.66
--webserver_interface=127.19.77.66
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:42.685662 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 19999
I20240501 19:56:42.685746 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:42.688058 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.67:0
--local_ip_for_outbound_sockets=127.19.77.67
--webserver_interface=127.19.77.67
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:42.853147 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 20130
I20240501 19:56:42.853263 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:42.855283 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.68:0
--local_ip_for_outbound_sockets=127.19.77.68
--webserver_interface=127.19.77.68
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:43.009645 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 20261
I20240501 19:56:43.009747 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:43.011682 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.69:0
--local_ip_for_outbound_sockets=127.19.77.69
--webserver_interface=127.19.77.69
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:43.186606 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 20392
I20240501 19:56:43.186717 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:43.188915 19765 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvj08vk/build/debug/bin/kudu
/tmp/dist-test-taskvj08vk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.77.70:0
--local_ip_for_outbound_sockets=127.19.77.70
--webserver_interface=127.19.77.70
--webserver_port=0
--tserver_master_addrs=127.19.77.126:38343
--builtin_ntp_servers=127.19.77.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240501 19:56:43.354060 19765 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvj08vk/build/debug/bin/kudu as pid 20523
I20240501 19:56:43.354180 19765 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:43.368284 19765 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 19:56:43.488941 19765 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 19:56:43.490500 19765 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 58edb0d892d341109f86898f8e7ec128 to finish bootstrapping
I20240501 19:56:43.496201 19765 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9885c443cd6547e58217a8c53134a553 to finish bootstrapping
I20240501 19:56:43.500681 19765 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 66f761d906974cf3b4b38ef208eb63c2 to finish bootstrapping
I20240501 19:56:43.505180 19765 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9364168ae3af41f68474eb18a015ff17 to finish bootstrapping
I20240501 19:56:43.509673 19765 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 64391e918b7541dabaa3a08f8c1f6a48 to finish bootstrapping
I20240501 19:56:43.514449 19765 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e077de997504bf38b67a133340c4efb to finish bootstrapping
I20240501 19:56:43.519232 19765 test_util.cc:268] Using random seed: -490716932
F20240501 19:56:52.767838 21303 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0fb64cb33cf24bce84591348caa59521.queue_time_us, tablet-download-pool-0fb64cb33cf24bce84591348caa59521.queue_time_us) (tablet-download-pool-0fb64cb33cf24bce84591348caa59521.run_cpu_time_us, tablet-download-pool-0fb64cb33cf24bce84591348caa59521.run_cpu_time_us) (tablet-download-pool-0fb64cb33cf24bce84591348caa59521.run_wall_time_us, tablet-download-pool-0fb64cb33cf24bce84591348caa59521.run_wall_time_us) (tablet-download-pool-151114e083b7470498bf8736468082ac.queue_time_us, tablet-download-pool-151114e083b7470498bf8736468082ac.queue_time_us) (tablet-download-pool-151114e083b7470498bf8736468082ac.run_cpu_time_us, tablet-download-pool-151114e083b7470498bf8736468082ac.run_cpu_time_us) (tablet-download-pool-151114e083b7470498bf8736468082ac.run_wall_time_us, tablet-download-pool-151114e083b7470498bf8736468082ac.run_wall_time_us) (tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.queue_time_us, tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.queue_time_us) (tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.run_cpu_time_us, tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.run_cpu_time_us) (tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.run_wall_time_us, tablet-download-pool-1dd4da3441b44399b2a6190bffdf80bf.run_wall_time_us) (tablet-download-pool-26c8fa003a7546208fd527498d522a5d.queue_time_us, tablet-download-pool-26c8fa003a7546208fd527498d522a5d.queue_time_us) (tablet-download-pool-26c8fa003a7546208fd527498d522a5d.run_cpu_time_us, tablet-download-pool-26c8fa003a7546208fd527498d522a5d.run_cpu_time_us) (tablet-download-pool-26c8fa003a7546208fd527498d522a5d.run_wall_time_us, tablet-download-pool-26c8fa003a7546208fd527498d522a5d.run_wall_time_us) (tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.queue_time_us, tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.queue_time_us) (tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.run_cpu_time_us, tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.run_cpu_time_us) (tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.run_wall_time_us, tablet-download-pool-450b1aec6a1d49f7a271c5a62dbc1336.run_wall_time_us) (tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.queue_time_us, tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.queue_time_us) (tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.run_cpu_time_us, tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.run_cpu_time_us) (tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.run_wall_time_us, tablet-download-pool-4822c9eba3e84a5b92cb9a745f75f956.run_wall_time_us) (tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.queue_time_us, tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.queue_time_us) (tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.run_cpu_time_us, tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.run_cpu_time_us) (tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.run_wall_time_us, tablet-download-pool-5464255c1a43498f9492dfc9d70fd092.run_wall_time_us) (tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.queue_time_us, tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.queue_time_us) (tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.run_cpu_time_us, tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.run_cpu_time_us) (tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.run_wall_time_us, tablet-download-pool-7a3185130b2d41d9ba6b401343f61b11.run_wall_time_us) (tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.queue_time_us, tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.queue_time_us) (tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.run_cpu_time_us, tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.run_cpu_time_us) (tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.run_wall_time_us, tablet-download-pool-8d15b27a60e84831ba3fe7df30f56929.run_wall_time_us) (tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.queue_time_us, tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.queue_time_us) (tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.run_cpu_time_us, tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.run_cpu_time_us) (tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.run_wall_time_us, tablet-download-pool-90793a3a3b3b47148c2f3bb8931651be.run_wall_time_us) (tablet-download-pool-9106cd16f7cd46278571720c29cf896a.queue_time_us, tablet-download-pool-9106cd16f7cd46278571720c29cf896a.queue_time_us) (tablet-download-pool-9106cd16f7cd46278571720c29cf896a.run_cpu_time_us, tablet-download-pool-9106cd16f7cd46278571720c29cf896a.run_cpu_time_us) (tablet-download-pool-9106cd16f7cd46278571720c29cf896a.run_wall_time_us, tablet-download-pool-9106cd16f7cd46278571720c29cf896a.run_wall_time_us) (tablet-download-pool-a619a130df7847a8a83a5e090ddf7639.queue_time_us, tablet-download-pool-a619a130df7847a8a83a5e090ddf7639.queue_time_us) (tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.queue_time_us, tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.queue_time_us) (tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.run_cpu_time_us, tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.run_cpu_time_us) (tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.run_wall_time_us, tablet-download-pool-aeea9ff6bcfc4bf7a640790f30c3815a.run_wall_time_us) (tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.queue_time_us, tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.queue_time_us) (tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.run_cpu_time_us, tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.run_cpu_time_us) (tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.run_wall_time_us, tablet-download-pool-bfb95e057c8847acbebcc535b8c0ad7c.run_wall_time_us) (tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.queue_time_us, tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.queue_time_us) (tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.run_cpu_time_us, tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.run_cpu_time_us) (tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.run_wall_time_us, tablet-download-pool-ec1fc6245dc8464d828fc58d3bb43cd6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f7c51e85dcd google::LogMessage::Fail() at ??:0
@ 0x7f7c51e89b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7c51e857cc google::LogMessage::Flush() at ??:0
@ 0x7f7c51e86f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7c532ef044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f7c532d3f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f7c532d1efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f7c55e8477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7c55e8da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f7c55f0e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f7c55f224f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f7c55f0d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f7c55f1cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559ba2d5275c std::function<>::operator()() at ??:0
@ 0x7f7c532d6e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7c532d76ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7c532d903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559ba2d5275c std::function<>::operator()() at ??:0
@ 0x7f7c532c6807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7c53b046db start_thread at ??:0
@ 0x7f7c5103c71f clone at ??:0
W20240501 19:56:56.696446 21456 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:53.692208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.77.66:36725 (local address 127.0.0.1:58008)
0501 19:56:53.695974 (+ 3766us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:53.695985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:53.696150 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:56.695759 (+2999609us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.19.77.66:36725: received 0 of 34 requested bytes
Metrics: {"client-negotiator.queue_time_us":3716}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 13 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 6 | 0
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | HEALTHY | <none> | 1 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.66:36725: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.66:36725: received 0 of 34 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.68 | experimental | 127.19.77.68:42889
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.19.77.68:42889
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet e1aae74bb62f4c58b13b2bf60b061306 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): RUNNING
All reported replicas are:
A = 66f761d906974cf3b4b38ef208eb63c2
B = 9885c443cd6547e58217a8c53134a553
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | [config not available] | | |
C | A* B C | 1 | 3 | Yes
Tablet e6a56c0702fa4bd4a2ca80c95ea9d650 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
All reported replicas are:
A = 4e077de997504bf38b67a133340c4efb
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): RUNNING
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 67 | Yes
C | A* B C | 1 | 67 | Yes
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 2 | -1 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 2 | -1 | Yes
Tablet 9c19edfc38b84855b9f716a1f26cae99 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet fa995ca6685e43b1964c8f2b3c6cb171 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 6c6f425f093442d298d7b69088ef884c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet edcbedf0b0124cc1ba73005b58931ef1 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 344987a73dd64438baf1c1f0f4aff8a8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 3 | -1 | Yes
Tablet b637c44528ce44f8b74bb71b8dfa794b of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 2 | -1 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): TS unavailable [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 19 | 0 | 2 | 19
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 20
First Quartile | 20
Median | 21
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 124
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
F20240501 19:56:56.926947 21745 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-12be3adba6da40adb2dc0970fa998d79.queue_time_us, tablet-download-pool-12be3adba6da40adb2dc0970fa998d79.queue_time_us) (tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.queue_time_us, tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.queue_time_us) (tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.run_cpu_time_us, tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.run_cpu_time_us) (tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.run_wall_time_us, tablet-download-pool-15f9c4d88c7149ba9cfdfeae5916967c.run_wall_time_us) (tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.queue_time_us, tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.queue_time_us) (tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.run_cpu_time_us, tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.run_cpu_time_us) (tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.run_wall_time_us, tablet-download-pool-27dbaa24a8dc4b73bc01e89fa80d9103.run_wall_time_us) (tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.queue_time_us, tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.queue_time_us) (tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.run_cpu_time_us, tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.run_cpu_time_us) (tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.run_wall_time_us, tablet-download-pool-2a87ab50074640bda0b53361a892bfb0.run_wall_time_us) (tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.queue_time_us, tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.queue_time_us) (tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.run_cpu_time_us, tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.run_cpu_time_us) (tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.run_wall_time_us, tablet-download-pool-344987a73dd64438baf1c1f0f4aff8a8.run_wall_time_us) (tablet-download-pool-6c6f425f093442d298d7b69088ef884c.queue_time_us, tablet-download-pool-6c6f425f093442d298d7b69088ef884c.queue_time_us) (tablet-download-pool-6c6f425f093442d298d7b69088ef884c.run_cpu_time_us, tablet-download-pool-6c6f425f093442d298d7b69088ef884c.run_cpu_time_us) (tablet-download-pool-6c6f425f093442d298d7b69088ef884c.run_wall_time_us, tablet-download-pool-6c6f425f093442d298d7b69088ef884c.run_wall_time_us) (tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.queue_time_us, tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.queue_time_us) (tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.run_cpu_time_us, tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.run_cpu_time_us) (tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.run_wall_time_us, tablet-download-pool-77f5cc98907b4e61ae1c3dfa16a5a4de.run_wall_time_us) (tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.queue_time_us, tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.queue_time_us) (tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.run_cpu_time_us, tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.run_cpu_time_us) (tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.run_wall_time_us, tablet-download-pool-9c19edfc38b84855b9f716a1f26cae99.run_wall_time_us) (tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.queue_time_us, tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.queue_time_us) (tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.run_cpu_time_us, tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.run_cpu_time_us) (tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.run_wall_time_us, tablet-download-pool-b637c44528ce44f8b74bb71b8dfa794b.run_wall_time_us) (tablet-download-pool-b930392bd0a746b881768d4399ef7969.queue_time_us, tablet-download-pool-b930392bd0a746b881768d4399ef7969.queue_time_us) (tablet-download-pool-b930392bd0a746b881768d4399ef7969.run_cpu_time_us, tablet-download-pool-b930392bd0a746b881768d4399ef7969.run_cpu_time_us) (tablet-download-pool-b930392bd0a746b881768d4399ef7969.run_wall_time_us, tablet-download-pool-b930392bd0a746b881768d4399ef7969.run_wall_time_us) (tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.queue_time_us, tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.queue_time_us) (tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.run_cpu_time_us, tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.run_cpu_time_us) (tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.run_wall_time_us, tablet-download-pool-b9cdbb8d46d144e0b7b8a38c9f695f0d.run_wall_time_us) (tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.queue_time_us, tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.queue_time_us) (tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.run_cpu_time_us, tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.run_cpu_time_us) (tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.run_wall_time_us, tablet-download-pool-cfd24af7c06548e0b08ebd0ba8c0df1c.run_wall_time_us) (tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.queue_time_us, tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.queue_time_us) (tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.run_cpu_time_us, tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.run_cpu_time_us) (tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.run_wall_time_us, tablet-download-pool-edcbedf0b0124cc1ba73005b58931ef1.run_wall_time_us) (tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.queue_time_us, tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.queue_time_us) (tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.run_cpu_time_us, tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.run_cpu_time_us) (tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.run_wall_time_us, tablet-download-pool-fa995ca6685e43b1964c8f2b3c6cb171.run_wall_time_us) (tablet-download-pool-fef379a728394e13b9566de78d4b2c17.queue_time_us, tablet-download-pool-fef379a728394e13b9566de78d4b2c17.queue_time_us) (tablet-download-pool-fef379a728394e13b9566de78d4b2c17.run_cpu_time_us, tablet-download-pool-fef379a728394e13b9566de78d4b2c17.run_cpu_time_us) (tablet-download-pool-fef379a728394e13b9566de78d4b2c17.run_wall_time_us, tablet-download-pool-fef379a728394e13b9566de78d4b2c17.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fb2a5f36dcd google::LogMessage::Fail() at ??:0
@ 0x7fb2a5f3ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb2a5f367cc google::LogMessage::Flush() at ??:0
@ 0x7fb2a5f37f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb2a73a0044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fb2a7384f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fb2a7382efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fb2a9f3577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fb2a9f3ea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240501 19:56:57.073390 20687 meta_cache.cc:302] tablet 6c6f425f093442d298d7b69088ef884c: replica 58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745) has failed: Network error: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111)
@ 0x7fb2a9fbf76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fb2a9fd34f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fb2a9fbe226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fb2a9fcdc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b0e2d1a75c std::function<>::operator()() at ??:0
@ 0x7fb2a7387e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb2a73886ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb2a738a03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b0e2d1a75c std::function<>::operator()() at ??:0
@ 0x7fb2a7377807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb2a7bb56db start_thread at ??:0
@ 0x7fb2a50ed71f clone at ??:0
W20240501 19:56:57.329849 20689 connection.cc:535] client connection to 127.19.77.68:42889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet e1aae74bb62f4c58b13b2bf60b061306 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
All reported replicas are:
A = 66f761d906974cf3b4b38ef208eb63c2
B = 9364168ae3af41f68474eb18a015ff17
C = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 6 | Yes
B | [config not available] | | |
C | A* B C | 1 | 6 | Yes
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 70 | Yes
B | A* B C | 3 | 70 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 68 | Yes
C | A B* C D~ | 2 | 68 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 60 | Yes
C | A B* C | 1 | 60 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 65 | Yes
C | A B* C | 4 | 65 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 59 | Yes
C | A B* C D~ | 1 | 59 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 63 | Yes
C | A B C* | 1 | 63 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 65 | Yes
C | A B* C | 3 | 65 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 63 | Yes
C | A B* C | 1 | 63 | Yes
Tablet 9c19edfc38b84855b9f716a1f26cae99 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 66 | Yes
B | A B* C | 1 | 66 | Yes
C | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | -1 | Yes
C | A B C* | 3 | -1 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 61 | Yes
C | A B* C | 1 | 61 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 57 | Yes
C | A B* C D~ | 1 | 57 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet fa995ca6685e43b1964c8f2b3c6cb171 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 60 | Yes
B | A B* C | 1 | 60 | Yes
C | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 59 | Yes
C | A B* C | 1 | 59 | Yes
Tablet 6c6f425f093442d298d7b69088ef884c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 68 | Yes
B | A B* C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 60 | Yes
C | A B* C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet edcbedf0b0124cc1ba73005b58931ef1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 66 | Yes
B | A B* C | 2 | 66 | Yes
C | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 59 | Yes
C | A B* C D~ | 1 | 59 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet 344987a73dd64438baf1c1f0f4aff8a8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 66 | Yes
B | A* B C | 1 | 66 | Yes
C | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 62 | Yes
C | A B* C D~ | 3 | 62 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 57 | Yes
C | A B C* | 1 | 57 | Yes
Tablet b637c44528ce44f8b74bb71b8dfa794b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 54 | Yes
B | A B* C | 1 | 54 | Yes
C | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 58 | Yes
C | A B* C D~ | 2 | 58 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 59 | Yes
C | A B* C | 1 | 59 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 61 | Yes
C | A B* C D~ | 1 | 61 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 1 | 0 | 39 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 20
Median | 21
Third Quartile | 21
Maximum | 39
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | 39
Small | 58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | 13
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet e1aae74bb62f4c58b13b2bf60b061306 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
All reported replicas are:
A = 66f761d906974cf3b4b38ef208eb63c2
B = 9364168ae3af41f68474eb18a015ff17
C = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 6 | Yes
B | [config not available] | | |
C | A* B C | 1 | 6 | Yes
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 70 | Yes
B | A* B C | 3 | 70 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 68 | Yes
C | A B* C D~ | 2 | 68 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 60 | Yes
C | A B* C | 1 | 60 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 65 | Yes
C | A B* C | 4 | 65 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 59 | Yes
C | A B* C D~ | 1 | 59 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 63 | Yes
C | A B C* | 1 | 63 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 65 | Yes
C | A B* C | 3 | 65 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 63 | Yes
C | A B* C | 1 | 63 | Yes
Tablet 9c19edfc38b84855b9f716a1f26cae99 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 66 | Yes
B | A B* C | 1 | 66 | Yes
C | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | -1 | Yes
C | A B C* | 3 | -1 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 61 | Yes
C | A B* C | 1 | 61 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 57 | Yes
C | A B* C D~ | 1 | 57 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet fa995ca6685e43b1964c8f2b3c6cb171 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 60 | Yes
B | A B* C | 1 | 60 | Yes
C | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 59 | Yes
C | A B* C | 1 | 59 | Yes
Tablet 6c6f425f093442d298d7b69088ef884c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 68 | Yes
B | A B* C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 60 | Yes
C | A B* C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet edcbedf0b0124cc1ba73005b58931ef1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 66 | Yes
B | A B* C | 2 | 66 | Yes
C | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet 344987a73dd64438baf1c1f0f4aff8a8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 66 | Yes
B | A* B C | 1 | 66 | Yes
C | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 65 | Yes
C | A B* C | 3 | 65 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 57 | Yes
C | A B C* | 1 | 57 | Yes
Tablet b637c44528ce44f8b74bb71b8dfa794b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 54 | Yes
B | A B* C | 1 | 54 | Yes
C | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 58 | Yes
C | A B* C D~ | 2 | 58 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 59 | Yes
C | A B* C | 1 | 59 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 1 | 0 | 39 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 20
Median | 21
Third Quartile | 21
Maximum | 36
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | 36
Small | 58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | 13
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet e1aae74bb62f4c58b13b2bf60b061306 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [NONVOTER]
All reported replicas are:
A = 66f761d906974cf3b4b38ef208eb63c2
B = 9364168ae3af41f68474eb18a015ff17
C = 64391e918b7541dabaa3a08f8c1f6a48
D = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 7 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 7 | Yes
D | A* B C D | 1 | 8 | Yes
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 77 | Yes
B | A* B C D~ | 3 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 63 | Yes
C | A B* C D~ | 1 | 63 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 69 | Yes
C | A B* C D~ | 4 | 69 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 65 | Yes
C | A B C* D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 70 | Yes
C | A B* C D~ | 3 | 70 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 9c19edfc38b84855b9f716a1f26cae99 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 1 | 74 | Yes
B | A B* D | 1 | 74 | Yes
C | [config not available] | | |
D | A B* D | 1 | 74 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 71 | Yes
C | A B C* D~ | 3 | 71 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 61 | Yes
C | A B* C | 1 | 61 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet fa995ca6685e43b1964c8f2b3c6cb171 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 1 | 68 | Yes
B | A B* D | 1 | 68 | Yes
C | [config not available] | | |
D | A B* D | 1 | 68 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 62 | Yes
C | A B* C D~ | 1 | 62 | Yes
D | [config not available] | | |
Tablet 6c6f425f093442d298d7b69088ef884c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 2 | 76 | No
B | A B* C D | 2 | 75 | Yes
C | [config not available] | | |
D | A B* C D | 2 | 75 | No
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 69 | Yes
C | A B* C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet edcbedf0b0124cc1ba73005b58931ef1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 2 | 69 | Yes
B | A B* C D~ | 2 | 69 | Yes
C | [config not available] | | |
D | A B* C D~ | 2 | 69 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 68 | Yes
C | A B C* D~ | 2 | 68 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 344987a73dd64438baf1c1f0f4aff8a8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet 344987a73dd64438baf1c1f0f4aff8a8: Tablet copy aborted
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 65 | Yes
C | A B* C | 3 | 65 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 62 | Yes
C | A B C* D~ | 1 | 62 | Yes
D | [config not available] | | |
Tablet b637c44528ce44f8b74bb71b8dfa794b of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 64391e918b7541dabaa3a08f8c1f6a48
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 1 | 61 | Yes
B | A B* D | 1 | 61 | Yes
C | [config not available] | | |
D | A B* D | 1 | 61 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 62 | Yes
C | A B* C | 2 | 62 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 62 | Yes
C | A B* C D~ | 1 | 62 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 64 | Yes
C | A B C* D~ | 2 | 64 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 1 | 0 | 39 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 27
Third Quartile | 28
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 151
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 77 | Yes
B | A* B C D~ | 3 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 69 | Yes
C | A B* C D~ | 4 | 69 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 75 | Yes
C | A B* C | 3 | 75 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 75 | Yes
C | A B C* | 3 | 75 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 72 | Yes
C | A B* C | 1 | 72 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 67 | Yes
C | A B* C D~ | 3 | 67 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 25
Third Quartile | 27
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 146
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 79 | Yes
B | A* B C | 3 | 79 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 71 | Yes
C | A B* C | 4 | 71 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 75 | Yes
C | A B* C | 3 | 75 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 75 | Yes
C | A B C* | 3 | 75 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 72 | Yes
C | A B* C | 1 | 72 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 65 | Yes
C | A B* C D~ | 1 | 65 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 67 | Yes
C | A B* C D~ | 3 | 67 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 79 | Yes
B | A* B C | 3 | 79 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 79 | Yes
C | A B C* D~ | 1 | 79 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 71 | Yes
C | A B* C | 4 | 71 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 69 | Yes
C | A B* C | 1 | 69 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 79 | Yes
C | A B* C D~ | 3 | 79 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 79 | Yes
C | A B C* D~ | 3 | 79 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 72 | Yes
C | A B* C | 1 | 72 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C D~ | 1 | 78 | No
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 71 | Yes
C | A B* C | 3 | 71 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 65 | Yes
C | A B* C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 29
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | A* B C D~ | 3 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 79 | Yes
C | A B C* D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 76 | Yes
C | A B* C D~ | 4 | 76 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 69 | Yes
C | A B* C | 1 | 69 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 73 | Yes
C | A B C* D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 79 | Yes
C | A B* C D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 77 | Yes
C | A B* C | 2 | 77 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 79 | Yes
C | A B C* D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 76 | Yes
C | A B C* D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 71 | Yes
C | A B* C | 3 | 71 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 68 | Yes
C | A B* C | 2 | 68 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 72 | Yes
C | A B C* D~ | 2 | 72 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | A* B C D~ | 3 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 79 | Yes
C | A B C* D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 76 | Yes
C | A B* C D~ | 4 | 76 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 69 | Yes
C | A B* C | 1 | 69 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 73 | Yes
C | A B C* D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 79 | Yes
C | A B* C D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 77 | Yes
C | A B* C | 2 | 77 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 79 | Yes
C | A B C* D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 76 | Yes
C | A B C* D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 71 | Yes
C | A B* C | 3 | 71 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 68 | Yes
C | A B* C | 2 | 68 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 72 | Yes
C | A B C* D~ | 2 | 72 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | A* B C D~ | 3 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 82 | Yes
C | A B C* | 1 | 82 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 79 | Yes
C | A B* C | 4 | 79 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 82 | Yes
C | A B* C | 3 | 82 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 78 | Yes
C | A B* C | 1 | 78 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 83 | Yes
C | A B C* | 3 | 83 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 77 | Yes
C | A B* C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 75 | Yes
C | A B* C D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 75 | Yes
C | A B* C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 78 | Yes
C | A B C* | 2 | 78 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 90 | Yes
B | A* B C | 3 | 90 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 82 | Yes
C | A B C* | 1 | 82 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 79 | Yes
C | A B* C | 4 | 79 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 82 | Yes
C | A B* C | 3 | 82 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 78 | Yes
C | A B* C | 1 | 78 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 83 | Yes
C | A B C* | 3 | 83 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 77 | Yes
C | A B* C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 75 | Yes
C | A B* C D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 75 | Yes
C | A B* C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 75 | Yes
C | A B* C D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 78 | Yes
C | A B C* | 2 | 78 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 90 | Yes
B | A* B C | 3 | 90 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 79 | Yes
C | A B* C | 4 | 79 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 79 | Yes
C | A B* C | 1 | 79 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 86 | Yes
C | A B* C | 1 | 86 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 88 | Yes
C | A B* C D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 84 | Yes
C | A B C* D~ | 2 | 84 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 80 | Yes
C | A B* C | 3 | 80 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 79 | Yes
C | A B* C | 1 | 79 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 81 | Yes
C | A B C* D~ | 2 | 81 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 138
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 93 | Yes
B | A* B C D~ | 3 | 93 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 83 | Yes
C | A B* C D~ | 4 | 83 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 79 | Yes
C | A B* C | 1 | 79 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 86 | Yes
C | A B* C | 1 | 86 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 88 | Yes
C | A B* C D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 84 | Yes
C | A B C* D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 80 | Yes
C | A B* C | 3 | 80 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 79 | Yes
C | A B* C | 1 | 79 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 81 | Yes
C | A B C* D~ | 2 | 81 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 93 | Yes
B | A* B C D~ | 3 | 93 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 93 | Yes
C | A B C* | 1 | 93 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 83 | Yes
C | A B* C D~ | 4 | 83 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 86 | Yes
C | A B* C | 1 | 86 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 93 | Yes
C | A B* C | 3 | 93 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 91 | Yes
C | A B C* | 3 | 91 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 81 | Yes
C | A B* C | 1 | 81 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 84 | Yes
C | A B C* D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 85 | Yes
C | A B* C D~ | 3 | 85 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 81 | Yes
C | A B C* D~ | 2 | 81 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 3 | 97 | Yes
B | A* B C | 3 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 93 | Yes
C | A B C* | 1 | 93 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 89 | Yes
C | A B* C | 4 | 89 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 88 | Yes
C | A B C* | 1 | 88 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 89 | Yes
C | A B* C D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 93 | Yes
C | A B* C | 3 | 93 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 89 | Yes
C | A B* C D~ | 2 | 89 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 91 | Yes
C | A B C* | 3 | 91 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 88 | Yes
C | A B C* | 2 | 88 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 85 | Yes
C | A B* C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 85 | Yes
C | A B C* | 1 | 85 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 93 | Yes
C | A B C* | 1 | 93 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 85 | Yes
C | A B C* | 2 | 85 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.66:36725, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 97 | Yes
B | A* B C | 3 | 97 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 93 | Yes
C | A B C* | 1 | 93 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 89 | Yes
C | A B* C | 4 | 89 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 88 | Yes
C | A B C* | 1 | 88 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 89 | Yes
C | A B* C D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 93 | Yes
C | A B* C | 3 | 93 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 89 | Yes
C | A B* C D~ | 2 | 89 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 91 | Yes
C | A B C* | 3 | 91 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 88 | Yes
C | A B C* | 2 | 88 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 79 | Yes
C | A B* C D~ | 1 | 79 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 85 | Yes
C | A B* C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 85 | Yes
C | A B C* | 1 | 85 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 93 | Yes
C | A B C* | 1 | 93 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 85 | Yes
C | A B C* | 2 | 85 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 97 | Yes
B | A* B C | 3 | 97 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 87 | Yes
C | A B* C D~ | 1 | 87 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 89 | Yes
C | A B* C | 4 | 89 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 88 | Yes
C | A B C* | 1 | 88 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 95 | Yes
C | A B* C D~ | 3 | 95 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 97 | Yes
C | A B C* D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 94 | Yes
C | A B C* D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 83 | Yes
C | A B* C | 1 | 83 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 89 | Yes
C | A B* C | 3 | 89 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 98 | Yes
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 86 | Yes
C | A B* C | 1 | 86 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 91 | Yes
C | A B C* D~ | 2 | 91 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 23
Third Quartile | 24
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 102 | Yes
B | A* B C D~ | 3 | 102 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 87 | Yes
C | A B* C D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 94 | Yes
C | A B* C D~ | 4 | 94 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 93 | Yes
C | A B C* D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 95 | Yes
C | A B* C D~ | 3 | 95 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 97 | Yes
C | A B C* D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 89 | Yes
C | A B* C | 1 | 89 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 94 | Yes
C | A B C* D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 83 | Yes
C | A B* C | 1 | 83 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 89 | Yes
C | A B* C | 3 | 89 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 98 | Yes
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 86 | Yes
C | A B* C | 1 | 86 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 91 | Yes
C | A B C* D~ | 2 | 91 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 102 | Yes
B | A* B C D~ | 3 | 102 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | Yes
C | A B C* | 1 | 102 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 87 | Yes
C | A B* C D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 94 | Yes
C | A B* C D~ | 4 | 94 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 93 | Yes
C | A B C* D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 99 | Yes
C | A B* C | 3 | 99 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 101 | Yes
C | A B C* | 3 | 101 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | No
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 94 | Yes
C | A B C* D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 87 | Yes
C | A B* C D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 94 | Yes
C | A B* C D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | No
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 96 | Yes
C | A B C* | 2 | 96 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 25
Third Quartile | 27
Maximum | 29
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 105 | Yes
B | A* B C | 3 | 105 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | Yes
C | A B C* | 1 | 102 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 99 | Yes
C | A B* C | 4 | 99 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 98 | Yes
C | A B C* | 1 | 98 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 97 | Yes
C | A B* C D~ | 1 | 97 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 99 | Yes
C | A B* C | 3 | 99 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 101 | Yes
C | A B C* | 3 | 101 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 97 | Yes
C | A B C* | 2 | 97 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 87 | Yes
C | A B* C D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 94 | Yes
C | A B* C D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | Yes
C | A B C* | 1 | 102 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 96 | Yes
C | A B C* | 2 | 96 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 105 | Yes
B | A* B C | 3 | 105 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | Yes
C | A B C* | 1 | 102 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 100 | Yes
C | A B* C | 1 | 100 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 99 | Yes
C | A B* C | 4 | 99 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 98 | Yes
C | A B C* | 1 | 98 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 97 | Yes
C | A B* C D~ | 1 | 97 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 103 | Yes
C | A B* C D~ | 3 | 103 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 101 | Yes
C | A B C* | 3 | 101 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 93 | Yes
C | A B* C D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 97 | Yes
C | A B C* | 2 | 97 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 99 | Yes
C | A B* C | 3 | 99 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 102 | Yes
C | A B C* | 1 | 102 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 96 | Yes
C | A B C* | 2 | 96 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 3 | 110 | Yes
B | A* B C D~ | 3 | 110 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 100 | Yes
C | A B* C | 1 | 100 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 104 | Yes
C | A B* C D~ | 4 | 104 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 103 | Yes
C | A B* C D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 105 | Yes
C | A B* C D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 104 | Yes
C | A B C* D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 103 | Yes
C | A B C* D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 99 | Yes
C | A B* C | 3 | 99 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 106 | Yes
C | A B C* D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 143
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 110 | Yes
B | A* B C D~ | 3 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 104 | Yes
C | A B* C D~ | 4 | 104 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 103 | Yes
C | A B* C D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 105 | Yes
C | A B* C D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 104 | Yes
C | A B C* D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 103 | Yes
C | A B C* D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 99 | Yes
C | A B* C | 3 | 99 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 106 | Yes
C | A B C* D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 110 | Yes
B | A* B C D~ | 3 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 102 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 110 | Yes
C | A B* C D~ | 1 | 110 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 104 | Yes
C | A B* C D~ | 4 | 104 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 108 | Yes
C | A B* C | 3 | 108 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 107 | Yes
C | A B C* | 3 | 107 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 107 | Yes
C | A B C* | 2 | 107 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 108 | Yes
C | A B* C | 1 | 108 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 104 | Yes
C | A B* C D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 106 | Yes
C | A B C* | 1 | 106 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 111 | Yes
C | A B C* | 1 | 111 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 108 | Yes
C | A B C* | 2 | 108 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 25
Third Quartile | 27
Maximum | 28
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 143
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 113 | Yes
B | A* B C | 3 | 113 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 109 | Yes
C | A B* C D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 102 | Yes
C | A B* C | 1 | 102 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 110 | Yes
C | A B* C D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 105 | Yes
C | A B* C | 4 | 105 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 108 | Yes
C | A B* C | 3 | 108 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 107 | Yes
C | A B C* | 3 | 107 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 101 | Yes
C | A B* C | 1 | 101 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 107 | Yes
C | A B C* | 2 | 107 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 108 | Yes
C | A B* C | 1 | 108 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 104 | Yes
C | A B* C D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 106 | Yes
C | A B C* | 1 | 106 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 111 | Yes
C | A B C* | 1 | 111 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 108 | Yes
C | A B C* | 2 | 108 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 113 | Yes
B | A* B C | 3 | 113 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 109 | Yes
C | A B* C D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 113 | Yes
C | A B C* D~ | 1 | 113 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 102 | Yes
C | A B* C | 1 | 102 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 110 | Yes
C | A B* C D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 105 | Yes
C | A B* C | 4 | 105 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 107 | Yes
C | A B C* | 2 | 107 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 106 | Yes
C | A B C* | 1 | 106 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 111 | Yes
C | A B C* | 1 | 111 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 108 | Yes
C | A B C* | 2 | 108 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 29
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 119 | Yes
B | A* B C D~ | 3 | 119 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 113 | Yes
C | A B C* D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 114 | Yes
C | A B* C | 1 | 114 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 111 | Yes
C | A B* C D~ | 4 | 111 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 112 | Yes
C | A B* C | 2 | 112 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 111 | Yes
C | A B C* D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 110 | Yes
C | A B C* D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 115 | Yes
C | A B C* D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 102 | Yes
C | A B* C | 2 | 102 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 110 | Yes
C | A B C* D~ | 2 | 110 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 119 | Yes
B | A* B C D~ | 3 | 119 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 113 | Yes
C | A B C* D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 114 | Yes
C | A B* C | 1 | 114 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 111 | Yes
C | A B* C D~ | 4 | 111 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 112 | Yes
C | A B* C | 2 | 112 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 111 | Yes
C | A B C* D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | No
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 110 | Yes
C | A B C* D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 115 | Yes
C | A B C* D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 102 | Yes
C | A B* C | 2 | 102 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 110 | Yes
C | A B C* D~ | 2 | 110 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 145
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 119 | Yes
B | A* B C D~ | 3 | 119 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 118 | Yes
C | A B C* | 1 | 118 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 114 | Yes
C | A B* C | 1 | 114 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 111 | Yes
C | A B* C D~ | 4 | 111 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* | 1 | 114 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 116 | Yes
C | A B* C | 3 | 116 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 117 | Yes
C | A B* C D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 116 | Yes
C | A B C* | 3 | 116 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 106 | Yes
C | A B* C D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 116 | Yes
C | A B C* | 2 | 116 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 109 | Yes
C | A B* C D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 143
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 122 | Yes
B | A* B C | 3 | 122 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 118 | Yes
C | A B C* | 1 | 118 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 116 | Yes
C | A B* C | 4 | 116 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 114 | Yes
C | A B C* | 1 | 114 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 116 | Yes
C | A B* C | 3 | 116 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 117 | Yes
C | A B* C D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 116 | Yes
C | A B C* | 3 | 116 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 106 | Yes
C | A B* C D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 116 | Yes
C | A B C* | 2 | 116 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 122 | Yes
C | A B* C | 1 | 122 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 109 | Yes
C | A B* C D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 116 | Yes
C | A B* C | 1 | 116 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 122 | Yes
B | A* B C | 3 | 122 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 116 | Yes
C | A B* C | 4 | 116 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 110 | Yes
C | A B* C | 1 | 110 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 114 | Yes
C | A B C* | 1 | 114 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 120 | Yes
C | A B* C D~ | 3 | 120 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 117 | Yes
C | A B* C D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 121 | Yes
C | A B C* D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 118 | No
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 116 | Yes
C | A B C* | 2 | 116 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 122 | Yes
C | A B* C | 1 | 122 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 109 | Yes
C | A B* C D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 116 | Yes
C | A B* C | 1 | 116 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 121 | Yes
C | A B C* D~ | 2 | 121 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.66:36725, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 127 | Yes
B | A* B C D~ | 3 | 127 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 126 | Yes
C | A B* C | 2 | 126 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 120 | Yes
C | A B* C D~ | 4 | 120 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 110 | Yes
C | A B* C | 1 | 110 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 116 | Yes
C | A B C* D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 122 | Yes
C | A B* C | 1 | 122 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 120 | Yes
C | A B* C D~ | 3 | 120 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 123 | Yes
C | A B* C D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 121 | Yes
C | A B C* D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 118 | Yes
C | A B* C | 1 | 118 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 116 | Yes
C | A B* C D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 119 | Yes
C | A B C* D~ | 2 | 119 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 113 | Yes
C | A B* C | 2 | 113 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 121 | Yes
C | A B C* D~ | 2 | 121 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 127 | Yes
B | A* B C D~ | 3 | 127 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 126 | Yes
C | A B* C | 2 | 126 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 120 | Yes
C | A B* C D~ | 4 | 120 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 116 | Yes
C | A B C* D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 122 | Yes
C | A B* C | 1 | 122 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 124 | Yes
C | A B* C | 3 | 124 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 123 | Yes
C | A B* C D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 121 | Yes
C | A B C* D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 118 | Yes
C | A B* C | 1 | 118 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 116 | Yes
C | A B* C D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 112 | Yes
C | A B* C | 1 | 112 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 119 | Yes
C | A B C* D~ | 2 | 119 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 113 | Yes
C | A B* C | 2 | 113 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 122 | Yes
C | A B* C D~ | 1 | 122 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | No
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 24
Third Quartile | 27
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 145
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 127 | Yes
B | A* B C D~ | 3 | 127 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 131 | Yes
C | A B* C D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 128 | Yes
C | A B C* | 1 | 128 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 124 | Yes
C | A B* C | 4 | 124 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 124 | Yes
C | A B* C | 3 | 124 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 123 | Yes
C | A B* C | 1 | 123 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 124 | Yes
C | A B C* | 3 | 124 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 119 | Yes
C | A B* C | 1 | 119 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 121 | Yes
C | A B C* | 2 | 121 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 123 | Yes
C | A B* C D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 122 | Yes
C | A B C* | 1 | 122 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 122 | Yes
C | A B* C D~ | 1 | 122 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 19
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 131 | Yes
B | A* B C | 3 | 131 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 131 | Yes
C | A B* C D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 128 | Yes
C | A B C* | 1 | 128 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 124 | Yes
C | A B* C | 4 | 124 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 117 | Yes
C | A B* C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 124 | Yes
C | A B* C | 3 | 124 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 123 | Yes
C | A B* C | 1 | 123 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 124 | Yes
C | A B C* | 3 | 124 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 119 | Yes
C | A B* C | 1 | 119 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 121 | Yes
C | A B C* | 2 | 121 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 123 | Yes
C | A B* C D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 122 | Yes
C | A B C* | 1 | 122 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 122 | Yes
C | A B* C D~ | 1 | 122 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 131 | Yes
B | A* B C | 3 | 131 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 131 | Yes
C | A B* C D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 130 | Yes
C | A B C* D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 128 | No
C | A B* C | 4 | 124 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 120 | Yes
C | A B C* | 1 | 120 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 128 | Yes
C | A B* C D~ | 3 | 128 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 125 | Yes
C | A B* C | 2 | 125 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 128 | Yes
C | A B C* D~ | 3 | 128 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 123 | Yes
C | A B* C D~ | 1 | 123 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 127 | Yes
C | A B C* D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 128 | Yes
C | A B* C | 3 | 128 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 23
Third Quartile | 23
Maximum | 27
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | 64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.66:36725, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 137 | Yes
B | A* B C D~ | 3 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 133 | Yes
C | A B* C | 2 | 133 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 130 | Yes
C | A B C* D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 128 | Yes
C | A B* C D~ | 4 | 128 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 124 | Yes
C | A B C* D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 128 | Yes
C | A B* C D~ | 3 | 128 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 125 | Yes
C | A B* C | 2 | 125 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 128 | Yes
C | A B C* D~ | 3 | 128 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 123 | Yes
C | A B* C D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 127 | Yes
C | A B C* D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 128 | Yes
C | A B* C | 3 | 128 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 130 | Yes
C | A B* C D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 137 | Yes
B | A* B C D~ | 3 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 133 | Yes
C | A B* C | 2 | 133 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 135 | Yes
C | A B C* | 1 | 135 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 128 | Yes
C | A B* C D~ | 4 | 128 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 124 | Yes
C | A B C* D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 134 | Yes
C | A B* C | 3 | 134 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | No
C | A B* C | 2 | 125 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 133 | Yes
C | A B C* | 3 | 133 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | No
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 123 | Yes
C | A B* C D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 132 | Yes
C | A B C* D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 130 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 130 | Yes
C | A B* C D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | No
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 27
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 147
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 142 | Yes
B | A* B C | 3 | 142 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 135 | Yes
C | A B C* | 1 | 135 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 131 | Yes
C | A B* C | 4 | 131 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 126 | Yes
C | A B C* | 1 | 126 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 134 | Yes
C | A B* C | 3 | 134 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 133 | Yes
C | A B C* | 3 | 133 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 141 | Yes
C | A B* C | 1 | 141 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 132 | Yes
C | A B C* | 2 | 132 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 130 | Yes
C | A B C* | 1 | 130 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 142 | Yes
B | A* B C | 3 | 142 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 135 | Yes
C | A B C* | 1 | 135 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 131 | Yes
C | A B* C | 4 | 131 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 126 | Yes
C | A B C* | 1 | 126 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 134 | Yes
C | A B* C | 3 | 134 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 133 | Yes
C | A B C* | 3 | 133 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 141 | Yes
C | A B* C | 1 | 141 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 132 | Yes
C | A B C* | 2 | 132 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 130 | Yes
C | A B C* | 1 | 130 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 142 | Yes
B | A* B C | 3 | 142 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 139 | Yes
C | A B C* D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 136 | Yes
C | A B* C D~ | 4 | 136 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 138 | Yes
C | A B* C D~ | 3 | 138 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 131 | Yes
C | A B* C | 2 | 131 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 135 | Yes
C | A B C* D~ | 3 | 135 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 132 | Yes
C | A B* C D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 138 | Yes
C | A B* C | 3 | 138 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 135 | Yes
C | A B C* D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 141 | Yes
C | A B C* D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 23
Third Quartile | 26
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 139
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.70:46009, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 145 | Yes
B | A* B C D~ | 3 | 145 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 139 | Yes
C | A B C* D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 136 | Yes
C | A B* C D~ | 4 | 136 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 132 | No
C | A B* C | 1 | 129 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 138 | Yes
C | A B* C D~ | 3 | 138 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 131 | Yes
C | A B* C | 2 | 131 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 135 | Yes
C | A B C* D~ | 3 | 135 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 132 | Yes
C | A B* C D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 138 | Yes
C | A B* C | 3 | 138 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 135 | Yes
C | A B C* D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 141 | Yes
C | A B C* D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 145 | Yes
B | A* B C D~ | 3 | 145 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 140 | No
C | A B* C D~ | 4 | 136 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 132 | Yes
C | A B* C D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 141 | Yes
C | A B* C | 3 | 141 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 139 | Yes
C | A B C* | 3 | 139 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | No
C | A B* C D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 138 | Yes
C | A B* C | 1 | 138 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 141 | Yes
C | A B C* | 2 | 141 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 129 | Yes
C | A B* C D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 142 | Yes
C | A B* C D~ | 3 | 142 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 146 | Yes
C | A B C* | 1 | 146 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 137 | Yes
C | A B* C D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 142 | Yes
C | A B C* | 2 | 142 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 26
Third Quartile | 27
Maximum | 28
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 149 | Yes
B | A* B C | 3 | 149 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 140 | Yes
C | A B* C | 4 | 140 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 132 | Yes
C | A B* C D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 136 | Yes
C | A B C* | 1 | 136 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 141 | Yes
C | A B* C | 3 | 141 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 139 | Yes
C | A B C* | 3 | 139 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 138 | Yes
C | A B* C | 1 | 138 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 141 | Yes
C | A B C* | 2 | 141 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 129 | Yes
C | A B* C D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 142 | Yes
C | A B* C D~ | 3 | 142 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 146 | Yes
C | A B C* | 1 | 146 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 137 | Yes
C | A B* C D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 142 | Yes
C | A B C* | 2 | 142 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.67:41937, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 149 | Yes
B | A* B C | 3 | 149 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 140 | Yes
C | A B* C | 4 | 140 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 136 | Yes
C | A B C* | 1 | 136 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 146 | Yes
C | A B* C | 3 | 141 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 139 | Yes
C | A B C* | 3 | 139 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 141 | Yes
C | A B C* | 2 | 141 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 147 | Yes
C | A B* C | 3 | 147 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 146 | Yes
C | A B C* | 1 | 146 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 142 | Yes
C | A B C* | 2 | 142 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 28
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 3 | 156 | Yes
B | A* B C D~ | 3 | 156 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 150 | Yes
C | A B* C | 2 | 150 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 145 | Yes
C | A B* C D~ | 4 | 145 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 146 | Yes
C | A B* C D~ | 3 | 146 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 143 | Yes
C | A B C* D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 155 | Yes
C | A B* C D~ | 1 | 155 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 143 | Yes
C | A B C* D~ | 2 | 143 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 147 | Yes
C | A B* C | 3 | 147 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 144 | Yes
C | A B C* D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 146 | Yes
C | A B C* D~ | 2 | 146 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 139
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.67:41937, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 156 | Yes
B | A* B C D~ | 3 | 156 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 150 | Yes
C | A B* C | 2 | 150 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 145 | Yes
C | A B* C D~ | 4 | 145 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 143 | Yes
C | A B* C D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 146 | Yes
C | A B* C D~ | 3 | 146 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 143 | Yes
C | A B C* D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 155 | Yes
C | A B* C D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 143 | Yes
C | A B C* D~ | 2 | 143 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 135 | Yes
C | A B* C | 1 | 135 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 147 | Yes
C | A B* C | 3 | 147 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 144 | Yes
C | A B C* D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 146 | Yes
C | A B C* D~ | 2 | 146 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 145
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 156 | Yes
B | A* B C D~ | 3 | 156 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 154 | Yes
C | A B* C D~ | 2 | 154 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 152 | Yes
C | A B C* | 1 | 152 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 149 | Yes
C | A B* C | 4 | 149 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 143 | Yes
C | A B* C D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 156 | Yes
C | A B* C D~ | 1 | 156 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 149 | Yes
C | A B* C | 3 | 149 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 148 | Yes
C | A B* C D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 147 | Yes
C | A B C* | 3 | 147 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 155 | Yes
C | A B* C D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 143 | Yes
C | A B* C D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 148 | Yes
C | A B* C | 1 | 148 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 147 | Yes
C | A B C* | 2 | 147 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 151 | Yes
C | A B* C D~ | 3 | 151 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 155 | Yes
C | A B C* | 1 | 155 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 153 | Yes
C | A B C* | 2 | 153 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 29
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 160 | Yes
B | A* B C | 3 | 160 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 154 | Yes
C | A B* C D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 152 | Yes
C | A B C* | 1 | 152 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 160 | Yes
C | A B* C D~ | 1 | 160 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 149 | Yes
C | A B* C | 4 | 149 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 143 | Yes
C | A B* C D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 156 | Yes
C | A B* C D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 149 | Yes
C | A B* C | 3 | 149 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 148 | Yes
C | A B* C D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 147 | Yes
C | A B C* | 3 | 147 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 160 | Yes
C | A B* C | 1 | 160 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 143 | Yes
C | A B* C D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 148 | Yes
C | A B* C | 1 | 148 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 147 | Yes
C | A B C* | 2 | 147 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 151 | Yes
C | A B* C D~ | 3 | 151 | Yes
D | [config not available] | | |
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 155 | Yes
C | A B C* | 1 | 155 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 153 | Yes
C | A B C* | 2 | 153 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.69:41613, tserver@127.19.77.66:36725, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 160 | Yes
B | A* B C | 3 | 160 | Yes
C | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 154 | Yes
C | A B* C D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 152 | Yes
C | A B C* | 1 | 152 | Yes
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 160 | Yes
C | A B* C D~ | 1 | 160 | Yes
D | [config not available] | | |
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 149 | Yes
C | A B* C | 4 | 149 | Yes
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 156 | Yes
C | A B* C D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 154 | Yes
C | A B* C D~ | 3 | 154 | Yes
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 148 | Yes
C | A B* C D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 152 | No
C | A B C* D~ | 3 | 152 | Yes
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 160 | Yes
C | A B* C | 1 | 160 | Yes
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
D | [config not available] | | |
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
D | [config not available] | | |
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 147 | Yes
C | A B C* | 2 | 147 | Yes
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 154 | Yes
C | A B* C | 3 | 154 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 155 | Yes
C | A B C* | 1 | 155 | Yes
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 153 | Yes
C | A B C* | 2 | 153 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 27
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 131
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d25e844ae4f14fdea53a8a2950cfbe3f | 127.19.77.126:38343 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.19.77.84:35145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4e077de997504bf38b67a133340c4efb | 127.19.77.70:46009 | HEALTHY | <none> | 6 | 0
64391e918b7541dabaa3a08f8c1f6a48 | 127.19.77.69:41613 | HEALTHY | <none> | 14 | 0
66f761d906974cf3b4b38ef208eb63c2 | 127.19.77.67:41937 | HEALTHY | <none> | 7 | 0
9885c443cd6547e58217a8c53134a553 | 127.19.77.66:36725 | HEALTHY | <none> | 14 | 0
58edb0d892d341109f86898f8e7ec128 | 127.19.77.65:40745 | UNAVAILABLE | <none> | n/a | n/a
9364168ae3af41f68474eb18a015ff17 | 127.19.77.68:42889 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.19.77.65:40745: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.65:40745: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.19.77.68:42889: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.19.77.68:42889: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
local_ip_for_outbound_sockets | 127.19.77.66 | experimental | 127.19.77.66:36725
local_ip_for_outbound_sockets | 127.19.77.67 | experimental | 127.19.77.67:41937
local_ip_for_outbound_sockets | 127.19.77.69 | experimental | 127.19.77.69:41613
local_ip_for_outbound_sockets | 127.19.77.70 | experimental | 127.19.77.70:46009
never_fsync | true | unsafe,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.19.77.66:36725
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.19.77.67:41937
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.19.77.69:41613
server_dump_info_path | /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.19.77.70:46009
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.19.77.84:35145 | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
time_source | builtin | 127.19.77.70:46009, 127.19.77.69:41613, 127.19.77.67:41937, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.19.77.126:38343, tserver@127.19.77.70:46009, tserver@127.19.77.69:41613, and 2 other server(s)
Tablet Summary
Tablet ed78666484da4654b51b2933b1ee5e3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9885c443cd6547e58217a8c53134a553
B = 66f761d906974cf3b4b38ef208eb63c2
C = 9364168ae3af41f68474eb18a015ff17
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 162 | Yes
B | A* B C D~ | 3 | 162 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet b9cdbb8d46d144e0b7b8a38c9f695f0d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 159 | Yes
C | A B* C | 2 | 159 | Yes
Tablet 5464255c1a43498f9492dfc9d70fd092 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 157 | Yes
C | A B C* D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet 67e698c1f92d4772bd9c71011782275c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet 26c8fa003a7546208fd527498d522a5d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 15f9c4d88c7149ba9cfdfeae5916967c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 165 | Yes
C | A B* C | 1 | 165 | Yes
Tablet d2f1e155a7474a05b9f7b9131f71dc6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 152 | Yes
C | A B* C D~ | 4 | 152 | Yes
D | [config not available] | | |
Tablet 77f5cc98907b4e61ae1c3dfa16a5a4de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet a619a130df7847a8a83a5e090ddf7639 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet cfd24af7c06548e0b08ebd0ba8c0df1c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 159 | Yes
C | A B* C | 1 | 159 | Yes
Tablet 0ff089a8953d4603830f410f3e7f28ee of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 154 | Yes
C | A B* C D~ | 3 | 154 | Yes
D | [config not available] | | |
Tablet 2a87ab50074640bda0b53361a892bfb0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 151 | Yes
C | A B* C | 2 | 151 | Yes
Tablet ec1fc6245dc8464d828fc58d3bb43cd6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
D | [config not available] | | |
Tablet 9106cd16f7cd46278571720c29cf896a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 157 | Yes
C | A B* C D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet b9f21cbedee1402d989ba79c5723e35a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING [LEADER]
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable [NONVOTER]
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
D = 9364168ae3af41f68474eb18a015ff17
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 152 | Yes
C | A B C* D~ | 3 | 152 | Yes
D | [config not available] | | |
Tablet 0fb64cb33cf24bce84591348caa59521 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 165 | Yes
C | A B* C D~ | 1 | 165 | Yes
D | [config not available] | | |
Tablet b930392bd0a746b881768d4399ef7969 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet 450b1aec6a1d49f7a271c5a62dbc1336 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 149 | Yes
C | A B* C D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet 151114e083b7470498bf8736468082ac of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
Tablet 7a3185130b2d41d9ba6b401343f61b11 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet fef379a728394e13b9566de78d4b2c17 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 4822c9eba3e84a5b92cb9a745f75f956 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
D | [config not available] | | |
Tablet bba66a025d7b4977b7bce975560e13e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 152 | Yes
C | A B C* D~ | 2 | 152 | Yes
D | [config not available] | | |
Tablet 12be3adba6da40adb2dc0970fa998d79 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 144 | Yes
C | A B* C | 1 | 144 | Yes
Tablet 8d15b27a60e84831ba3fe7df30f56929 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 154 | Yes
C | A B* C D~ | 1 | 154 | Yes
D | [config not available] | | |
Tablet 90793a3a3b3b47148c2f3bb8931651be of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 157 | Yes
C | A B* C D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet 0e13d83d3e474d45b23ad9ff3fa6bcbb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 154 | Yes
C | A B* C | 3 | 154 | Yes
Tablet 1dd4da3441b44399b2a6190bffdf80bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 152 | Yes
C | A B C* D~ | 1 | 152 | Yes
D | [config not available] | | |
Tablet aeea9ff6bcfc4bf7a640790f30c3815a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 159 | Yes
C | A B C* D~ | 1 | 159 | Yes
D | [config not available] | | |
Tablet 27dbaa24a8dc4b73bc01e89fa80d9103 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet bfb95e057c8847acbebcc535b8c0ad7c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING [LEADER]
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
Tablet ef3c809ef5e6466e9ec1cff3a779fe24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable
9885c443cd6547e58217a8c53134a553 (127.19.77.66:36725): RUNNING [LEADER]
66f761d906974cf3b4b38ef208eb63c2 (127.19.77.67:41937): RUNNING
All reported replicas are:
A = 58edb0d892d341109f86898f8e7ec128
B = 9885c443cd6547e58217a8c53134a553
C = 66f761d906974cf3b4b38ef208eb63c2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet 21fc4de45cdb4ff292ec4e33c7b6d162 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9364168ae3af41f68474eb18a015ff17 (127.19.77.68:42889): TS unavailable
64391e918b7541dabaa3a08f8c1f6a48 (127.19.77.69:41613): RUNNING
4e077de997504bf38b67a133340c4efb (127.19.77.70:46009): RUNNING [LEADER]
58edb0d892d341109f86898f8e7ec128 (127.19.77.65:40745): TS unavailable [NONVOTER]
All reported replicas are:
A = 9364168ae3af41f68474eb18a015ff17
B = 64391e918b7541dabaa3a08f8c1f6a48
C = 4e077de997504bf38b67a133340c4efb
D = 58edb0d892d341109f86898f8e7ec128
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 157 | Yes
C | A B C* D~ | 2 | 157 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 7 | 0 | 33 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 21
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
E20240501 19:57:19.789208 19765 external_mini_cluster.cc:943] Process with UUID 58edb0d892d341109f86898f8e7ec128 has crashed
E20240501 19:57:19.789299 19765 external_mini_cluster.cc:943] Process with UUID 9364168ae3af41f68474eb18a015ff17 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 19:57:20.725250 19765 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvj08vk/build/debug/bin/kudu with pid 19999
I20240501 19:57:20.734261 19765 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvj08vk/build/debug/bin/kudu with pid 20130
I20240501 19:57:20.741267 19765 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvj08vk/build/debug/bin/kudu with pid 20392
I20240501 19:57:20.747824 19765 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvj08vk/build/debug/bin/kudu with pid 20523
I20240501 19:57:20.754142 19765 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvj08vk/build/debug/bin/kudu with pid 19777
2024-05-01T19:57:20Z chronyd exiting
I20240501 19:57:20.771457 19765 test_util.cc:175] -----------------------------------------------
I20240501 19:57:20.771499 19765 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvj08vk/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714593402107785-19765-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (38655 ms)
[----------] 1 test from RaftConsensusStressITest (38655 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (38655 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240501 19:57:20.771939 19765 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 7 messages since previous log ~23 seconds ago