[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:01:32.731284 5494 test_util.cc:268] Using random seed: 173985670
I20240502 14:01:32.737305 5494 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:01:32.737356 5494 ts_itest-base.cc:113] --------------
I20240502 14:01:32.737367 5494 ts_itest-base.cc:114] 6 tablet servers
I20240502 14:01:32.737378 5494 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:01:32.737386 5494 ts_itest-base.cc:116] --------------
2024-05-02T14:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:01:32Z Disabled control of system clock
I20240502 14:01:32.746202 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.93.190:36985
--webserver_interface=127.5.93.190
--webserver_port=0
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.93.190:36985
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=40 with env {}
I20240502 14:01:32.910533 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 5506
I20240502 14:01:32.910655 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:01:32.962064 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.129:0
--local_ip_for_outbound_sockets=127.5.93.129
--webserver_interface=127.5.93.129
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.148124 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 5597
I20240502 14:01:33.148221 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:01:33.150489 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.130:0
--local_ip_for_outbound_sockets=127.5.93.130
--webserver_interface=127.5.93.130
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.315025 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 5728
I20240502 14:01:33.315138 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:01:33.316970 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.131:0
--local_ip_for_outbound_sockets=127.5.93.131
--webserver_interface=127.5.93.131
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.471052 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 5860
I20240502 14:01:33.471144 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:01:33.472986 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.132:0
--local_ip_for_outbound_sockets=127.5.93.132
--webserver_interface=127.5.93.132
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.637436 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 5991
I20240502 14:01:33.637544 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:01:33.639407 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.133:0
--local_ip_for_outbound_sockets=127.5.93.133
--webserver_interface=127.5.93.133
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.803790 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 6125
I20240502 14:01:33.803879 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:01:33.805816 5494 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
/tmp/dist-test-taskOVdOnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.93.134:0
--local_ip_for_outbound_sockets=127.5.93.134
--webserver_interface=127.5.93.134
--webserver_port=0
--tserver_master_addrs=127.5.93.190:36985
--builtin_ntp_servers=127.5.93.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 14:01:33.990657 5494 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu as pid 6256
I20240502 14:01:33.990759 5494 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:01:33.993319 5494 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:01:34.075994 5494 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:01:34.077503 5494 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2eab0c629f5f4d6caf6d77a8198d9980 to finish bootstrapping
I20240502 14:01:34.082680 5494 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b0f058493ec44c7fb0f6465be65cf09b to finish bootstrapping
I20240502 14:01:34.087363 5494 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 915d43a567bf44dbaae045d9ff6c36d4 to finish bootstrapping
I20240502 14:01:34.092100 5494 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2222cc707c1244e19c62a6fdb81bcc93 to finish bootstrapping
I20240502 14:01:34.096287 5494 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d633a34c7e594a35a858e82ca29625b9 to finish bootstrapping
I20240502 14:01:34.100015 5494 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 87d5e5fccb564f69a9743f927f192070 to finish bootstrapping
I20240502 14:01:34.104182 5494 test_util.cc:268] Using random seed: 175358579
F20240502 14:01:42.819998 6998 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0c5069cd4ea843e1be42c086c29f686f.queue_time_us, tablet-download-pool-0c5069cd4ea843e1be42c086c29f686f.queue_time_us) (tablet-download-pool-205f2c275b34407bacd959718206d056.queue_time_us, tablet-download-pool-205f2c275b34407bacd959718206d056.queue_time_us) (tablet-download-pool-205f2c275b34407bacd959718206d056.run_cpu_time_us, tablet-download-pool-205f2c275b34407bacd959718206d056.run_cpu_time_us) (tablet-download-pool-205f2c275b34407bacd959718206d056.run_wall_time_us, tablet-download-pool-205f2c275b34407bacd959718206d056.run_wall_time_us) (tablet-download-pool-2e10d42b8e9743279084825805738340.queue_time_us, tablet-download-pool-2e10d42b8e9743279084825805738340.queue_time_us) (tablet-download-pool-2e10d42b8e9743279084825805738340.run_cpu_time_us, tablet-download-pool-2e10d42b8e9743279084825805738340.run_cpu_time_us) (tablet-download-pool-2e10d42b8e9743279084825805738340.run_wall_time_us, tablet-download-pool-2e10d42b8e9743279084825805738340.run_wall_time_us) (tablet-download-pool-2edceb95cad545e797a20774f0272705.queue_time_us, tablet-download-pool-2edceb95cad545e797a20774f0272705.queue_time_us) (tablet-download-pool-2edceb95cad545e797a20774f0272705.run_cpu_time_us, tablet-download-pool-2edceb95cad545e797a20774f0272705.run_cpu_time_us) (tablet-download-pool-2edceb95cad545e797a20774f0272705.run_wall_time_us, tablet-download-pool-2edceb95cad545e797a20774f0272705.run_wall_time_us) (tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.queue_time_us, tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.queue_time_us) (tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.run_cpu_time_us, tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.run_cpu_time_us) (tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.run_wall_time_us, tablet-download-pool-4489cb8195c846b1a8c36fdd772b1b29.run_wall_time_us) (tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.queue_time_us, tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.queue_time_us) (tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.run_cpu_time_us, tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.run_cpu_time_us) (tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.run_wall_time_us, tablet-download-pool-561d6b28d8c34ba0ab1a6a7c317d94ef.run_wall_time_us) (tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.queue_time_us, tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.queue_time_us) (tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.run_cpu_time_us, tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.run_cpu_time_us) (tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.run_wall_time_us, tablet-download-pool-6254bed1d7194976a8ed417c0f8a0474.run_wall_time_us) (tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.queue_time_us, tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.queue_time_us) (tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.run_cpu_time_us, tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.run_cpu_time_us) (tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.run_wall_time_us, tablet-download-pool-a936f270d8bc4350b07ce3933ee8035a.run_wall_time_us) (tablet-download-pool-b949e8e50227461992887659182c4498.queue_time_us, tablet-download-pool-b949e8e50227461992887659182c4498.queue_time_us) (tablet-download-pool-b949e8e50227461992887659182c4498.run_cpu_time_us, tablet-download-pool-b949e8e50227461992887659182c4498.run_cpu_time_us) (tablet-download-pool-b949e8e50227461992887659182c4498.run_wall_time_us, tablet-download-pool-b949e8e50227461992887659182c4498.run_wall_time_us) (tablet-download-pool-d95b1095734546a089095b73ebd0b065.queue_time_us, tablet-download-pool-d95b1095734546a089095b73ebd0b065.queue_time_us) (tablet-download-pool-d95b1095734546a089095b73ebd0b065.run_cpu_time_us, tablet-download-pool-d95b1095734546a089095b73ebd0b065.run_cpu_time_us) (tablet-download-pool-d95b1095734546a089095b73ebd0b065.run_wall_time_us, tablet-download-pool-d95b1095734546a089095b73ebd0b065.run_wall_time_us) (tablet-download-pool-e196377875a6401c80925c33730779d6.queue_time_us, tablet-download-pool-e196377875a6401c80925c33730779d6.queue_time_us) (tablet-download-pool-e196377875a6401c80925c33730779d6.run_cpu_time_us, tablet-download-pool-e196377875a6401c80925c33730779d6.run_cpu_time_us) (tablet-download-pool-e196377875a6401c80925c33730779d6.run_wall_time_us, tablet-download-pool-e196377875a6401c80925c33730779d6.run_wall_time_us) (tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.queue_time_us, tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.queue_time_us) (tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.run_cpu_time_us, tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.run_cpu_time_us) (tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.run_wall_time_us, tablet-download-pool-e1a53539d67a451aa5c2166e52d28f96.run_wall_time_us) (tablet-download-pool-eabbb051e0bc4e979572214455d681f2.queue_time_us, tablet-download-pool-eabbb051e0bc4e979572214455d681f2.queue_time_us) (tablet-download-pool-eabbb051e0bc4e979572214455d681f2.run_cpu_time_us, tablet-download-pool-eabbb051e0bc4e979572214455d681f2.run_cpu_time_us) (tablet-download-pool-eabbb051e0bc4e979572214455d681f2.run_wall_time_us, tablet-download-pool-eabbb051e0bc4e979572214455d681f2.run_wall_time_us) (tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.queue_time_us, tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.queue_time_us) (tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.run_cpu_time_us, tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.run_cpu_time_us) (tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.run_wall_time_us, tablet-download-pool-eeb1766eed8a4752ad4207cee52bf123.run_wall_time_us) (tablet-download-pool-f34db69da7564922a4a11293aa1db952.queue_time_us, tablet-download-pool-f34db69da7564922a4a11293aa1db952.queue_time_us) (tablet-download-pool-f34db69da7564922a4a11293aa1db952.run_cpu_time_us, tablet-download-pool-f34db69da7564922a4a11293aa1db952.run_cpu_time_us) (tablet-download-pool-f34db69da7564922a4a11293aa1db952.run_wall_time_us, tablet-download-pool-f34db69da7564922a4a11293aa1db952.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f7bdc7a4dcd google::LogMessage::Fail() at ??:0
@ 0x7f7bdc7a8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7bdc7a47cc google::LogMessage::Flush() at ??:0
@ 0x7f7bdc7a5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7bddc0e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f7bddbf2f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f7bddbf0efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f7be07a377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7be07aca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f7be082d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f7be08414f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f7be082c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f7be083bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556a02ec675c std::function<>::operator()() at ??:0
@ 0x7f7bddbf5e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7bddbf66ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7bddbf803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556a02ec675c std::function<>::operator()() at ??:0
@ 0x7f7bddbe5807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7bde4236db start_thread at ??:0
@ 0x7f7bdb95b71f clone at ??:0
W20240502 14:01:46.823994 7242 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:43.819532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.93.131:33019 (local address 127.0.0.1:47734)
0502 14:01:43.820078 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:43.820091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:43.820165 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:46.823343 (+3003178us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.5.93.131:33019: received 0 of 34 requested bytes
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":69,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | HEALTHY | <none> | 0 | 0
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 12 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 9 | 0
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.131:33019: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.131:33019: received 0 of 34 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.132 | experimental | 127.5.93.132:42447
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.93.132:42447
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
time_source | builtin | 127.5.93.132:42447, 127.5.93.134:40355, 127.5.93.131:33019, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.132:42447, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 2 | -1 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 1 | 62 | Yes
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 1 | 64 | Yes
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 66 | Yes
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 54 | Yes
B | A B C* | 1 | 54 | Yes
C | A B C* | 1 | 54 | Yes
D | [config not available] | | |
Tablet 84f16536bac444a491b956aceb815538 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 63 | Yes
B | A B C* | 1 | 63 | Yes
C | A B C* | 1 | 63 | Yes
D | [config not available] | | |
Tablet 49fdd482e2804c34beec6d564e1d0dd5 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 2 | 64 | Yes
B | A B* C | 2 | 64 | Yes
C | A B* C | 2 | 64 | Yes
D | [config not available] | | |
Tablet f676f8cbdc5c4b2f86f202902f366968 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | -1 | Yes
C | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 67 | Yes
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 2 | 65 | Yes
B | A B* C | 2 | 65 | Yes
C | A B* C | 2 | 65 | Yes
D | [config not available] | | |
Tablet a000d081d39e40fc8c85fd72161b677a of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 62 | Yes
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
D | [config not available] | | |
Tablet 7c51e118fc2242489e346cf9818215c8 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 62 | Yes
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
D | [config not available] | | |
Tablet ef111b2f4b8a469388443876a39afca3 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 481d3656deaa4648bee4d9e36dad36b5 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 62 | Yes
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
D | [config not available] | | |
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 66 | Yes
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
D | [config not available] | | |
Tablet b88725f5e578494798789254837d9abf of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 6254bed1d7194976a8ed417c0f8a0474 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 64 | Yes
B | A B* C | 1 | 64 | Yes
C | [config not available] | | |
Tablet 3434d0ebc65648ebae1c8175c60931cb of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 1 | 65 | Yes
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | -1 | Yes
C | [config not available] | | |
Tablet 2edceb95cad545e797a20774f0272705 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 61 | Yes
B | A B* C | 1 | 61 | Yes
C | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 1 | 68 | Yes
B | A B C* | 1 | 68 | Yes
C | A B C* | 1 | 68 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 1 | 67 | Yes
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
D | [config not available] | | |
Tablet 0ee6685b24224aa58d7495969f7a8d42 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): TS unavailable
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 1 | 63 | Yes
B | A B* C | 1 | 63 | Yes
C | A B* C | 1 | 63 | Yes
D | [config not available] | | |
Tablet a936f270d8bc4350b07ce3933ee8035a of table 'TestTable' is conflicted: 3 replicas' active configs disagree with the leader master's
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2222cc707c1244e19c62a6fdb81bcc93
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A* B C | 1 | 3 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 40 | 2 | 0 | 2 | 36
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 19
First Quartile | 20
Median | 21
Third Quartile | 21
Maximum | 39
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 39
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
F20240502 14:01:47.067008 7514 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.queue_time_us, tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.queue_time_us) (tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.run_cpu_time_us, tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.run_cpu_time_us) (tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.run_wall_time_us, tablet-download-pool-0ee6685b24224aa58d7495969f7a8d42.run_wall_time_us) (tablet-download-pool-2a358caed74e43bcbf274fd5c557504a.queue_time_us, tablet-download-pool-2a358caed74e43bcbf274fd5c557504a.queue_time_us) (tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.queue_time_us, tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.queue_time_us) (tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.run_cpu_time_us, tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.run_cpu_time_us) (tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.run_wall_time_us, tablet-download-pool-3434d0ebc65648ebae1c8175c60931cb.run_wall_time_us) (tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.queue_time_us, tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.queue_time_us) (tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.run_cpu_time_us, tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.run_cpu_time_us) (tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.run_wall_time_us, tablet-download-pool-481d3656deaa4648bee4d9e36dad36b5.run_wall_time_us) (tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.queue_time_us, tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.queue_time_us) (tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.run_cpu_time_us, tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.run_cpu_time_us) (tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.run_wall_time_us, tablet-download-pool-49fdd482e2804c34beec6d564e1d0dd5.run_wall_time_us) (tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.queue_time_us, tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.queue_time_us) (tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.run_cpu_time_us, tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.run_cpu_time_us) (tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.run_wall_time_us, tablet-download-pool-54a945b92ec64e59a82d1981bbfac19e.run_wall_time_us) (tablet-download-pool-7c51e118fc2242489e346cf9818215c8.queue_time_us, tablet-download-pool-7c51e118fc2242489e346cf9818215c8.queue_time_us) (tablet-download-pool-7c51e118fc2242489e346cf9818215c8.run_cpu_time_us, tablet-download-pool-7c51e118fc2242489e346cf9818215c8.run_cpu_time_us) (tablet-download-pool-7c51e118fc2242489e346cf9818215c8.run_wall_time_us, tablet-download-pool-7c51e118fc2242489e346cf9818215c8.run_wall_time_us) (tablet-download-pool-84f16536bac444a491b956aceb815538.queue_time_us, tablet-download-pool-84f16536bac444a491b956aceb815538.queue_time_us) (tablet-download-pool-84f16536bac444a491b956aceb815538.run_cpu_time_us, tablet-download-pool-84f16536bac444a491b956aceb815538.run_cpu_time_us) (tablet-download-pool-84f16536bac444a491b956aceb815538.run_wall_time_us, tablet-download-pool-84f16536bac444a491b956aceb815538.run_wall_time_us) (tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.queue_time_us, tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.queue_time_us) (tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.run_cpu_time_us, tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.run_cpu_time_us) (tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.run_wall_time_us, tablet-download-pool-a000d081d39e40fc8c85fd72161b677a.run_wall_time_us) (tablet-download-pool-a9e127f655494633907fc7a733fdbf94.queue_time_us, tablet-download-pool-a9e127f655494633907fc7a733fdbf94.queue_time_us) (tablet-download-pool-a9e127f655494633907fc7a733fdbf94.run_cpu_time_us, tablet-download-pool-a9e127f655494633907fc7a733fdbf94.run_cpu_time_us) (tablet-download-pool-a9e127f655494633907fc7a733fdbf94.run_wall_time_us, tablet-download-pool-a9e127f655494633907fc7a733fdbf94.run_wall_time_us) (tablet-download-pool-b88725f5e578494798789254837d9abf.queue_time_us, tablet-download-pool-b88725f5e578494798789254837d9abf.queue_time_us) (tablet-download-pool-b88725f5e578494798789254837d9abf.run_cpu_time_us, tablet-download-pool-b88725f5e578494798789254837d9abf.run_cpu_time_us) (tablet-download-pool-b88725f5e578494798789254837d9abf.run_wall_time_us, tablet-download-pool-b88725f5e578494798789254837d9abf.run_wall_time_us) (tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.queue_time_us, tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.queue_time_us) (tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.run_cpu_time_us, tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.run_cpu_time_us) (tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.run_wall_time_us, tablet-download-pool-d6ab3ba8d89f437aace4599ecabb1490.run_wall_time_us) (tablet-download-pool-ef111b2f4b8a469388443876a39afca3.queue_time_us, tablet-download-pool-ef111b2f4b8a469388443876a39afca3.queue_time_us) (tablet-download-pool-ef111b2f4b8a469388443876a39afca3.run_cpu_time_us, tablet-download-pool-ef111b2f4b8a469388443876a39afca3.run_cpu_time_us) (tablet-download-pool-ef111b2f4b8a469388443876a39afca3.run_wall_time_us, tablet-download-pool-ef111b2f4b8a469388443876a39afca3.run_wall_time_us) (tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.queue_time_us, tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.queue_time_us) (tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.run_cpu_time_us, tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.run_cpu_time_us) (tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.run_wall_time_us, tablet-download-pool-f676f8cbdc5c4b2f86f202902f366968.run_wall_time_us) (tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.queue_time_us, tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.queue_time_us) (tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.run_cpu_time_us, tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.run_cpu_time_us) (tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.run_wall_time_us, tablet-download-pool-fb9d88ae250d45379395b1f33bf5fce9.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f645e6d2dcd google::LogMessage::Fail() at ??:0
@ 0x7f645e6d6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f645e6d27cc google::LogMessage::Flush() at ??:0
@ 0x7f645e6d3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f645fb3c044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f645fb20f98 kudu::ThreadPool::ThreadPool() at ??:0
W20240502 14:01:47.129055 6419 meta_cache.cc:302] tablet d6ab3ba8d89f437aace4599ecabb1490: replica 2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563) has failed: Network error: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111)
@ 0x7f645fb1eefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f64626d177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f64626daa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f646275b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f646276f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f646275a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f6462769c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5581f2f5275c std::function<>::operator()() at ??:0
@ 0x7f645fb23e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f645fb246ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f645fb2603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5581f2f5275c std::function<>::operator()() at ??:0
@ 0x7f645fb13807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f64603516db start_thread at ??:0
@ 0x7f645d88971f clone at ??:0
W20240502 14:01:47.497431 5501 connection.cc:535] client connection to 127.5.93.132:42447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:47.499656 7652 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:47.402195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.93.132:42447 (local address 127.0.0.1:56516)
0502 14:01:47.402698 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:47.402712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:47.402807 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:47.499546 (+ 96739us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.93.132:42447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":419}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet a936f270d8bc4350b07ce3933ee8035a of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 66 | Yes
C | A B* C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 60 | Yes
C | A B C* D~ | 2 | 60 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 64 | Yes
C | A B C* D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 54 | Yes
C | A B C* | 3 | 54 | Yes
Tablet 84f16536bac444a491b956aceb815538 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 64 | Yes
B | A* B C | 2 | 64 | Yes
C | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 63 | Yes
C | A B C* | 1 | 63 | Yes
Tablet 49fdd482e2804c34beec6d564e1d0dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 63 | Yes
B | A B* C | 1 | 63 | Yes
C | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 64 | Yes
C | A B* C | 2 | 64 | Yes
Tablet f676f8cbdc5c4b2f86f202902f366968 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 71 | Yes
B | A* B C | 3 | 71 | Yes
C | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 63 | Yes
C | A B C* D~ | 1 | 63 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 65 | Yes
C | A B* C | 2 | 65 | Yes
Tablet a000d081d39e40fc8c85fd72161b677a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 68 | Yes
B | A B* C | 1 | 68 | Yes
C | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 56 | Yes
C | A B C* D~ | 1 | 56 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet 7c51e118fc2242489e346cf9818215c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 63 | Yes
B | A* B C | 2 | 63 | Yes
C | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet ef111b2f4b8a469388443876a39afca3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 70 | Yes
B | A* B C | 2 | 70 | Yes
C | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet 481d3656deaa4648bee4d9e36dad36b5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 65 | Yes
B | A* B C | 2 | 65 | Yes
C | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 62 | Yes
C | A B* C | 2 | 62 | Yes
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 66 | Yes
B | A* B C | 2 | 66 | Yes
C | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet b88725f5e578494798789254837d9abf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 58 | Yes
B | A* B C | 1 | 58 | Yes
C | [config not available] | | |
Tablet 3434d0ebc65648ebae1c8175c60931cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | A* B C | 1 | 65 | Yes
C | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 66 | Yes
C | A B C* D~ | 3 | 66 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 61 | Yes
C | A B C* D~ | 1 | 61 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 68 | Yes
C | A B C* | 1 | 68 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 64 | Yes
C | A B C* D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 0ee6685b24224aa58d7495969f7a8d42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 68 | Yes
B | A* B C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 63 | Yes
C | A B* C | 1 | 63 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 2 | 0 | 38 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 20
Median | 21
Third Quartile | 22
Maximum | 39
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | 39
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet a936f270d8bc4350b07ce3933ee8035a of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 66 | Yes
C | A B* C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 60 | Yes
C | A B C* D~ | 2 | 60 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 64 | Yes
C | A B C* D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 54 | Yes
C | A B C* | 3 | 54 | Yes
Tablet 84f16536bac444a491b956aceb815538 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 64 | Yes
B | A* B C | 2 | 64 | Yes
C | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 63 | Yes
C | A B C* | 1 | 63 | Yes
Tablet 49fdd482e2804c34beec6d564e1d0dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 63 | Yes
B | A B* C | 1 | 63 | Yes
C | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 64 | Yes
C | A B* C | 2 | 64 | Yes
Tablet f676f8cbdc5c4b2f86f202902f366968 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 71 | Yes
B | A* B C | 3 | 71 | Yes
C | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 67 | Yes
C | A B C* | 1 | 67 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 63 | Yes
C | A B C* D~ | 1 | 63 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 65 | Yes
C | A B* C | 2 | 65 | Yes
Tablet a000d081d39e40fc8c85fd72161b677a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 68 | Yes
B | A B* C | 1 | 68 | Yes
C | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 56 | Yes
C | A B C* D~ | 1 | 56 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet 7c51e118fc2242489e346cf9818215c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 63 | Yes
B | A* B C | 2 | 63 | Yes
C | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet ef111b2f4b8a469388443876a39afca3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 70 | Yes
B | A* B C | 2 | 70 | Yes
C | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 62 | Yes
C | A B C* | 1 | 62 | Yes
Tablet 481d3656deaa4648bee4d9e36dad36b5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 65 | Yes
B | A* B C | 2 | 65 | Yes
C | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 62 | Yes
C | A B* C | 2 | 62 | Yes
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 66 | Yes
B | A* B C | 2 | 66 | Yes
C | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet b88725f5e578494798789254837d9abf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 58 | Yes
B | A* B C | 1 | 58 | Yes
C | [config not available] | | |
Tablet 3434d0ebc65648ebae1c8175c60931cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | A* B C | 1 | 65 | Yes
C | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 66 | Yes
C | A B C* D~ | 3 | 66 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 61 | Yes
C | A B C* D~ | 1 | 61 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 68 | Yes
C | A B C* | 1 | 68 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 64 | Yes
C | A B C* D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 0ee6685b24224aa58d7495969f7a8d42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 68 | Yes
B | A* B C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 63 | Yes
C | A B* C | 1 | 63 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 2 | 0 | 38 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 20
Median | 21
Third Quartile | 22
Maximum | 39
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | 39
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 74 | Yes
C | A B C* | 2 | 74 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 70 | Yes
C | A B* C D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 63 | Yes
C | A B C* | 2 | 63 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 70 | Yes
C | A B C* | 1 | 70 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 60 | Yes
C | A B C* D~ | 3 | 60 | Yes
Tablet 84f16536bac444a491b956aceb815538 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 70 | Yes
B | A* B C D~ | 2 | 70 | Yes
C | [config not available] | | |
D | A B C D~ | 2 | 70 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 68 | Yes
C | A B C* D~ | 1 | 68 | Yes
D | [config not available] | | |
Tablet 49fdd482e2804c34beec6d564e1d0dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 1 | 68 | Yes
B | A B* C D~ | 1 | 68 | Yes
C | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 68 | Yes
C | A B* C D~ | 2 | 68 | Yes
D | [config not available] | | |
Tablet f676f8cbdc5c4b2f86f202902f366968 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 76 | Yes
B | A* B C D~ | 3 | 76 | Yes
C | [config not available] | | |
D | A B C D~ | 3 | 76 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 68 | Yes
C | A B C* | 1 | 68 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
D | [config not available] | | |
Tablet a000d081d39e40fc8c85fd72161b677a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 73 | Yes
B | A B* C D~ | 1 | 73 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 73 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 60 | Yes
C | A B C* | 1 | 60 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 69 | Yes
C | A B C* D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 7c51e118fc2242489e346cf9818215c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 68 | Yes
B | A* B C D~ | 2 | 68 | Yes
C | [config not available] | | |
D | A B C D~ | 2 | 68 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* D~ | 1 | 67 | Yes
Tablet ef111b2f4b8a469388443876a39afca3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): missing [NONVOTER]
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 75 | Yes
B | A* B C D~ | 2 | 75 | Yes
C | [config not available] | | |
D | A B C D~ | 2 | 75 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 66 | Yes
C | A B C* D~ | 1 | 66 | Yes
Tablet 481d3656deaa4648bee4d9e36dad36b5 of table 'RemoveReplaceInCycle' is recovering: 1 on-going tablet copies
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: INITIALIZED
Data state: TABLET_DATA_COPYING
Last status: Tablet Copy: Replacing tablet superblock
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 72 | Yes
B | A* B C D~ | 2 | 72 | Yes
C | [config not available] | | |
D | A B C D~ | 2 | 72 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is recovering: 1 on-going tablet copies
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_COPYING
Last status: Remote error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: No such session: NO_SESSION: received error code Not found: Tablet Copy session with Session ID "d633a34c7e594a35a858e82ca29625b9-fb9d88ae250d45379395b1f33bf5fce9" not found from remote service
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 71 | Yes
B | A* B C | 2 | 71 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
Tablet b88725f5e578494798789254837d9abf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 63 | Yes
B | A* B C D~ | 1 | 63 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 63 | Yes
Tablet 3434d0ebc65648ebae1c8175c60931cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | A* B C D~ | 1 | 69 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 69 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 70 | Yes
C | A B C* | 3 | 70 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 65 | Yes
C | A B C* | 1 | 65 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 73 | Yes
C | A B C* D~ | 1 | 73 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 68 | Yes
C | A B C* | 2 | 68 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet 0ee6685b24224aa58d7495969f7a8d42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): missing [NONVOTER]
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 74 | Yes
B | A* B C D~ | 2 | 74 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
Tablet a936f270d8bc4350b07ce3933ee8035a of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet a936f270d8bc4350b07ce3933ee8035a: Tablet copy aborted
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2222cc707c1244e19c62a6fdb81bcc93
D = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 4 | Yes
B | A* B C D~ | 1 | 4 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 2 | 2 | 36 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 17
First Quartile | 21
Median | 22
Third Quartile | 30
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 142
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet a936f270d8bc4350b07ce3933ee8035a of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [NONVOTER]
All reported replicas are:
A = d633a34c7e594a35a858e82ca29625b9
B = 87d5e5fccb564f69a9743f927f192070
C = 2222cc707c1244e19c62a6fdb81bcc93
D = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B D | 1 | 6 | Yes
B | A* B D | 1 | 6 | Yes
C | [config not available] | | |
D | A* B D | 1 | 6 | Yes
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 72 | Yes
C | A B* C D~ | 2 | 72 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 76 | Yes
C | A B C* D~ | 2 | 76 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 70 | Yes
C | A B C* | 1 | 70 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 60 | Yes
C | A B C* D~ | 3 | 60 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 68 | Yes
C | A B C* | 1 | 71 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 68 | Yes
C | A B* C D~ | 2 | 68 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 70 | No
C | A B C* D~ | 1 | 70 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 64 | Yes
C | A B C* D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 69 | Yes
C | A B C* | 1 | 72 | No
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* | 1 | 70 | No
D | [config not available] | | |
Tablet ef111b2f4b8a469388443876a39afca3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [NONVOTER]
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B D | 2 | 78 | No
B | A* B D | 2 | 78 | Yes
C | [config not available] | | |
D | A* B C D | 2 | 77 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 66 | Yes
C | A B C* D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
D | [config not available] | | |
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 71 | Yes
B | A* B C | 2 | 71 | Yes
C | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 71 | Yes
C | A B* C D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 74 | Yes
C | A B C* D~ | 3 | 74 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 73 | Yes
C | A B C* D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 73 | Yes
C | A B C* D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 11 | 0 | 29 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 27
Third Quartile | 27
Maximum | 31
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 149
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 72 | Yes
C | A B* C D~ | 2 | 72 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 68 | Yes
C | A B* C | 1 | 68 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 76 | Yes
C | A B C* D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 62 | Yes
C | A B C* | 3 | 62 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 71 | Yes
C | A B C* | 1 | 71 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 71 | Yes
C | A B* C | 2 | 71 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 70 | Yes
C | A B C* D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 75 | Yes
C | A B* C | 2 | 75 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 64 | Yes
C | A B C* D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 72 | Yes
C | A B C* | 1 | 72 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 70 | Yes
C | A B C* | 1 | 70 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 73 | Yes
C | A B* C | 2 | 73 | Yes
Tablet fb9d88ae250d45379395b1f33bf5fce9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING [LEADER]
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [NONVOTER]
All reported replicas are:
A = b0f058493ec44c7fb0f6465be65cf09b
B = 915d43a567bf44dbaae045d9ff6c36d4
C = 2222cc707c1244e19c62a6fdb81bcc93
D = d633a34c7e594a35a858e82ca29625b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 74 | Yes
B | A* B C D~ | 2 | 74 | Yes
C | [config not available] | | |
D | A B C D~ | 2 | 74 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 74 | Yes
C | A B* C | 1 | 74 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 74 | Yes
C | A B C* D~ | 3 | 74 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 73 | Yes
C | A B C* D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 12 | 0 | 28 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 28
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 68 | Yes
C | A B* C | 1 | 68 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 76 | Yes
C | A B C* D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 72 | Yes
C | A B C* D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 62 | Yes
C | A B C* | 3 | 62 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 71 | Yes
C | A B C* | 1 | 71 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 71 | Yes
C | A B* C | 2 | 71 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 70 | Yes
C | A B C* D~ | 1 | 70 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 75 | Yes
C | A B* C | 2 | 75 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 72 | Yes
C | A B C* | 1 | 72 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 70 | Yes
C | A B C* | 1 | 70 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 73 | Yes
C | A B* C | 2 | 73 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 74 | Yes
C | A B* C | 1 | 74 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 78 | Yes
C | A B C* | 3 | 78 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 72 | Yes
C | A B C* | 1 | 72 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 75 | Yes
C | A B* C | 1 | 75 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 130
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 77 | Yes
C | A B* C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 68 | Yes
C | A B C* D~ | 3 | 68 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 77 | Yes
C | A B C* D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 77 | Yes
C | A B* C D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 78 | Yes
C | A B C* | 3 | 78 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 72 | Yes
C | A B C* | 1 | 72 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 81 | Yes
C | A B* C D~ | 1 | 81 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 77 | Yes
C | A B* C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 68 | Yes
C | A B C* D~ | 3 | 68 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 75 | Yes
C | A B C* | 1 | 75 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 77 | Yes
C | A B C* D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 77 | Yes
C | A B* C D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 78 | Yes
C | A B C* | 3 | 78 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 72 | Yes
C | A B C* | 1 | 72 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 81 | Yes
C | A B* C D~ | 1 | 81 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 75 | Yes
C | A B C* D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 68 | Yes
C | A B C* D~ | 3 | 68 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 81 | Yes
C | A B C* D~ | 1 | 81 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 77 | Yes
C | A B C* | 1 | 80 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* | 1 | 78 | No
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 74 | Yes
C | A B C* D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 77 | Yes
C | A B* C D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 80 | Yes
C | A B* C D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 82 | Yes
C | A B C* D~ | 3 | 82 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 82 | Yes
C | A B C* D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 81 | Yes
C | A B* C D~ | 1 | 81 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 25
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 146
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 75 | Yes
C | A B C* D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 80 | Yes
C | A B C* | 1 | 80 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 71 | Yes
C | A B C* | 3 | 71 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 81 | Yes
C | A B C* D~ | 1 | 81 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 80 | Yes
C | A B C* | 1 | 80 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 78 | Yes
C | A B C* | 1 | 78 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 77 | Yes
C | A B C* | 1 | 77 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 82 | Yes
C | A B C* D~ | 3 | 82 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 82 | Yes
C | A B C* D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 80 | Yes
C | A B C* | 1 | 80 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 80 | Yes
C | A B C* D~ | 1 | 80 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 71 | Yes
C | A B C* | 3 | 71 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 81 | Yes
C | A B C* D~ | 1 | 81 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 80 | Yes
C | A B C* | 1 | 80 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 78 | Yes
C | A B C* | 1 | 78 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 77 | Yes
C | A B C* | 1 | 77 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 87 | Yes
C | A B C* | 3 | 87 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 78 | Yes
C | A B C* D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 83 | Yes
C | A B C* | 1 | 83 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 82 | Yes
C | A B C* D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 26
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 131
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 91 | Yes
C | A B C* | 2 | 91 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 87 | Yes
C | A B C* D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 84 | Yes
C | A B C* | 1 | 84 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 75 | Yes
C | A B C* D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 90 | Yes
C | A B C* D~ | 1 | 90 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 88 | Yes
C | A B* C D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 78 | Yes
C | A B C* | 1 | 78 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 84 | Yes
C | A B C* D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 79 | Yes
C | A B* C | 2 | 79 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 87 | Yes
C | A B C* | 3 | 87 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 82 | Yes
C | A B C* | 1 | 82 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 91 | Yes
C | A B C* | 2 | 91 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 80 | Yes
C | A B C* | 2 | 80 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 87 | Yes
C | A B C* D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 84 | Yes
C | A B C* | 1 | 84 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 75 | Yes
C | A B C* D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 90 | Yes
C | A B C* D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 88 | Yes
C | A B* C D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 78 | Yes
C | A B C* | 1 | 78 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 84 | Yes
C | A B C* D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 87 | Yes
C | A B C* | 3 | 87 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 82 | Yes
C | A B C* | 1 | 82 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 97 | Yes
C | A B C* D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 83 | Yes
C | A B C* D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 87 | Yes
C | A B C* D~ | 1 | 87 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 84 | Yes
C | A B C* | 1 | 84 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 75 | Yes
C | A B C* D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 90 | Yes
C | A B C* D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 84 | Yes
C | A B C* D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* | 1 | 86 | No
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 91 | Yes
C | A B C* D~ | 3 | 91 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 89 | Yes
C | A B C* D~ | 1 | 89 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 91 | Yes
C | A B C* D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 25
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 146
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 97 | Yes
C | A B C* D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 83 | Yes
C | A B C* D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 80 | Yes
C | A B C* | 3 | 80 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 88 | Yes
C | A B C* | 1 | 88 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 86 | Yes
C | A B* C | 2 | 86 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 84 | Yes
C | A B C* D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 92 | Yes
C | A B C* | 1 | 92 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 91 | Yes
C | A B C* D~ | 3 | 91 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 91 | Yes
C | A B C* D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 97 | Yes
C | A B C* D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 83 | Yes
C | A B C* D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 80 | Yes
C | A B C* | 3 | 80 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 88 | Yes
C | A B C* | 1 | 88 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 86 | Yes
C | A B* C | 2 | 86 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 84 | Yes
C | A B C* D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 86 | Yes
C | A B C* | 1 | 86 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 92 | Yes
C | A B C* | 1 | 92 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 97 | Yes
C | A B C* | 3 | 97 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 85 | Yes
C | A B C* D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 91 | Yes
C | A B C* D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 26
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 131
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 100 | Yes
C | A B C* | 2 | 100 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 88 | Yes
C | A B C* D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 85 | Yes
C | A B C* D~ | 3 | 85 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 94 | Yes
C | A B C* D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 86 | Yes
C | A B* C | 2 | 86 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 89 | Yes
C | A B C* | 1 | 89 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 93 | Yes
C | A B C* D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 92 | Yes
C | A B C* D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 97 | Yes
C | A B C* | 3 | 97 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 95 | Yes
C | A B C* | 2 | 95 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 19
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 130
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 100 | Yes
C | A B C* | 2 | 100 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 85 | Yes
C | A B C* D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 94 | Yes
C | A B C* D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 89 | Yes
C | A B C* | 1 | 89 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 93 | Yes
C | A B C* D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 92 | Yes
C | A B C* D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 94 | Yes
C | A B* C D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 97 | Yes
C | A B C* | 3 | 97 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 90 | Yes
C | A B C* | 1 | 90 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 95 | Yes
C | A B C* | 2 | 95 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 100 | Yes
C | A B C* | 2 | 100 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 94 | Yes
C | A B C* | 1 | 94 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 85 | Yes
C | A B C* D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 94 | Yes
C | A B C* D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 92 | Yes
C | A B C* D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 93 | Yes
C | A B C* D~ | 1 | 93 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 92 | Yes
C | A B C* D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 94 | Yes
C | A B* C D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 96 | Yes
C | A B C* D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 100 | Yes
C | A B* C D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 100 | Yes
C | A B C* D~ | 3 | 100 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 95 | Yes
C | A B C* D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 99 | Yes
C | A B C* D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 24
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 147
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 98 | Yes
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 97 | Yes
C | A B* C | 2 | 97 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 101 | Yes
C | A B C* D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 104 | Yes
C | A B* C | 2 | 104 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 106 | Yes
C | A B* C | 2 | 106 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 92 | Yes
C | A B C* D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 96 | Yes
C | A B C* | 1 | 96 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 101 | Yes
C | A B C* | 1 | 101 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 100 | Yes
C | A B C* D~ | 3 | 100 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 95 | Yes
C | A B C* D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 103 | Yes
C | A B* C | 1 | 103 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 98 | Yes
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 97 | Yes
C | A B* C | 2 | 97 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 101 | Yes
C | A B C* D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 104 | Yes
C | A B* C | 2 | 104 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 106 | Yes
C | A B* C | 2 | 106 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 96 | Yes
C | A B C* | 1 | 96 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 101 | Yes
C | A B C* | 1 | 101 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 100 | Yes
C | A B C* D~ | 3 | 100 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 95 | Yes
C | A B C* D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 109 | Yes
C | A B C* | 2 | 109 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 98 | Yes
C | A B C* | 2 | 98 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 98 | Yes
C | A B C* D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 97 | Yes
C | A B* C | 2 | 97 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 110 | Yes
C | A B* C D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 106 | Yes
C | A B* C | 2 | 106 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 100 | Yes
C | A B C* D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 103 | Yes
C | A B C* | 3 | 103 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 104 | Yes
C | A B C* | 2 | 104 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 98 | Yes
C | A B* C | 1 | 98 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 20
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 109 | Yes
C | A B C* | 2 | 109 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 98 | Yes
C | A B C* | 2 | 98 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 103 | Yes
C | A B C* | 1 | 103 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 94 | Yes
C | A B C* D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 110 | Yes
C | A B* C D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 110 | Yes
C | A B* C D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 97 | Yes
C | A B C* | 1 | 97 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 100 | Yes
C | A B C* D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 100 | Yes
C | A B C* D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 103 | Yes
C | A B* C D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 110 | Yes
C | A B* C D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 103 | Yes
C | A B C* | 3 | 103 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 99 | Yes
C | A B C* | 1 | 99 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 104 | Yes
C | A B C* | 2 | 104 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 109 | Yes
C | A B C* | 2 | 109 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 103 | Yes
C | A B C* | 1 | 103 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 94 | Yes
C | A B C* D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 110 | Yes
C | A B* C D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 110 | Yes
C | A B* C D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 100 | Yes
C | A B C* D~ | 1 | 100 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 103 | Yes
C | A B* C D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 110 | Yes
C | A B* C D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 107 | Yes
C | A B C* D~ | 3 | 107 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 112 | Yes
C | A B C* D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 111 | Yes
C | A B C* | 1 | 111 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 108 | Yes
C | A B C* D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 100 | Yes
C | A B C* | 3 | 100 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 113 | Yes
C | A B C* | 1 | 113 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 112 | Yes
C | A B C* D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 114 | Yes
C | A B* C | 2 | 114 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 115 | Yes
C | A B* C | 2 | 115 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 107 | Yes
C | A B C* D~ | 3 | 107 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 112 | Yes
C | A B C* D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 111 | Yes
C | A B C* | 1 | 111 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 108 | Yes
C | A B C* D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 100 | Yes
C | A B C* | 3 | 100 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 113 | Yes
C | A B C* | 1 | 113 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 112 | Yes
C | A B C* D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 114 | Yes
C | A B* C | 2 | 114 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 115 | Yes
C | A B* C | 2 | 115 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 102 | Yes
C | A B C* D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 107 | Yes
C | A B C* D~ | 3 | 107 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 104 | Yes
C | A B C* D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 117 | Yes
C | A B* C | 2 | 117 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C | 1 | 104 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 119 | Yes
C | A B C* | 2 | 119 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 107 | Yes
C | A B C* | 2 | 107 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 115 | Yes
C | A B C* | 1 | 111 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 108 | Yes
C | A B C* D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 100 | Yes
C | A B C* | 3 | 100 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 113 | Yes
C | A B C* | 1 | 113 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 115 | Yes
C | A B* C | 2 | 115 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 106 | Yes
C | A B C* | 1 | 106 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 112 | Yes
C | A B C* D~ | 1 | 112 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 107 | Yes
C | A B C* | 1 | 107 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 110 | Yes
C | A B C* | 3 | 110 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 21
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 117 | Yes
C | A B* C | 2 | 117 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 119 | Yes
C | A B C* | 2 | 119 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 107 | Yes
C | A B C* | 2 | 107 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 115 | Yes
C | A B C* D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 112 | Yes
C | A B C* | 1 | 112 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 105 | Yes
C | A B C* D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 117 | Yes
C | A B C* D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 106 | Yes
C | A B C* | 1 | 106 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 112 | Yes
C | A B C* D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 111 | Yes
C | A B* C D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 110 | Yes
C | A B C* | 3 | 110 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 108 | Yes
C | A B* C D~ | 1 | 108 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 119 | Yes
C | A B C* | 2 | 119 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 111 | Yes
C | A B C* D~ | 2 | 111 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 115 | Yes
C | A B C* D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 112 | Yes
C | A B C* | 1 | 112 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 105 | Yes
C | A B C* D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 117 | Yes
C | A B C* D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 112 | Yes
C | A B C* D~ | 1 | 112 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 109 | Yes
C | A B C* D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 111 | Yes
C | A B* C D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 116 | Yes
C | A B C* D~ | 3 | 116 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* D~ | 2 | 120 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 112 | Yes
C | A B* C D~ | 1 | 112 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 144
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 122 | Yes
C | A B C* D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 111 | Yes
C | A B C* D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 119 | Yes
C | A B C* | 1 | 119 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 108 | Yes
C | A B C* | 3 | 108 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 118 | Yes
C | A B C* | 1 | 118 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 113 | Yes
C | A B* C | 2 | 113 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 114 | Yes
C | A B C* | 1 | 114 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 116 | Yes
C | A B C* D~ | 3 | 116 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 124 | Yes
C | A B C* | 1 | 124 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 113 | Yes
C | A B* C | 1 | 113 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 122 | Yes
C | A B C* D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 111 | Yes
C | A B C* D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 119 | Yes
C | A B C* | 1 | 119 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 108 | Yes
C | A B C* | 3 | 108 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 118 | Yes
C | A B C* | 1 | 118 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 113 | Yes
C | A B* C | 2 | 113 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 111 | Yes
C | A B C* D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 116 | Yes
C | A B C* | 1 | 116 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 114 | Yes
C | A B C* | 1 | 114 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 116 | Yes
C | A B C* D~ | 3 | 116 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 124 | Yes
C | A B C* | 1 | 124 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 128 | Yes
C | A B C* | 2 | 128 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 113 | Yes
C | A B* C | 2 | 113 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 122 | Yes
C | A B C* D~ | 1 | 122 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 114 | Yes
C | A B C* | 1 | 114 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 116 | Yes
C | A B* C | 2 | 116 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 121 | Yes
C | A B C* | 1 | 121 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 127 | Yes
C | A B* C | 1 | 127 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | A B C* | 3 | 120 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 119 | Yes
C | A B C* | 1 | 119 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* | 1 | 124 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 130 | Yes
C | A B* C D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 20
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 13
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 129
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 128 | Yes
C | A B C* | 2 | 128 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 115 | Yes
C | A B C* | 2 | 115 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 117 | Yes
C | A B* C D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 122 | Yes
C | A B C* D~ | 1 | 122 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 120 | Yes
C | A B C* | 3 | 120 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 119 | Yes
C | A B C* | 1 | 119 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 130 | Yes
C | A B* C D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 118 | Yes
C | A B* C D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 117 | Yes
C | A B* C D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 125 | Yes
C | A B C* D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 134 | Yes
C | A B* C | 2 | 134 | No
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 120 | Yes
C | A B C* D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 122 | Yes
C | A B C* D~ | 1 | 122 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 118 | Yes
C | A B C* D~ | 1 | 118 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 123 | Yes
C | A B C* D~ | 1 | 123 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 123 | Yes
C | A B C* D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 124 | Yes
C | A B C* D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 131 | Yes
C | A B C* D~ | 2 | 131 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 130 | Yes
C | A B* C D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 23
Third Quartile | 26
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 145
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 131 | Yes
C | A B* C | 1 | 131 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 116 | Yes
C | A B C* | 3 | 116 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 126 | Yes
C | A B C* | 1 | 126 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 122 | Yes
C | A B* C | 2 | 122 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 130 | Yes
C | A B C* | 1 | 130 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 134 | Yes
C | A B* C | 2 | 134 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 120 | Yes
C | A B C* D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 127 | Yes
C | A B C* | 1 | 127 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 128 | Yes
C | A B C* | 1 | 128 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 123 | Yes
C | A B C* D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 124 | Yes
C | A B C* D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 131 | Yes
C | A B C* D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 125 | Yes
C | A B* C | 1 | 125 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 131 | Yes
C | A B* C | 1 | 131 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 120 | Yes
C | A B C* | 2 | 125 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 116 | Yes
C | A B C* | 3 | 116 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 126 | Yes
C | A B C* | 1 | 126 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 122 | Yes
C | A B* C | 2 | 122 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 130 | Yes
C | A B C* | 1 | 130 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 134 | Yes
C | A B* C | 2 | 134 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 120 | Yes
C | A B C* D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 127 | Yes
C | A B C* | 1 | 127 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 124 | Yes
C | A B* C | 2 | 124 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 128 | Yes
C | A B C* | 1 | 128 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 123 | Yes
C | A B C* D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 131 | Yes
C | A B C* D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 125 | Yes
C | A B* C | 1 | 125 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 134 | Yes
C | A B* C | 2 | 134 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 127 | Yes
C | A B* C D~ | 1 | 127 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 125 | Yes
C | A B C* | 2 | 125 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 133 | Yes
C | A B C* | 1 | 133 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 121 | Yes
C | A B C* D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 122 | Yes
C | A B* C | 2 | 122 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 136 | Yes
C | A B C* D~ | 1 | 136 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 135 | Yes
C | A B C* | 1 | 135 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 133 | Yes
C | A B C* D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 128 | Yes
C | A B C* | 1 | 128 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 128 | Yes
C | A B C* | 3 | 128 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 136 | Yes
C | A B C* | 2 | 136 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 129 | Yes
C | A B* C D~ | 1 | 129 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 134 | Yes
C | A B* C | 2 | 134 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 127 | Yes
C | A B* C D~ | 1 | 127 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 125 | Yes
C | A B C* | 2 | 125 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 133 | Yes
C | A B C* | 1 | 133 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 121 | Yes
C | A B C* D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 126 | Yes
C | A B* C D~ | 2 | 126 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 136 | Yes
C | A B C* D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 135 | Yes
C | A B C* | 1 | 135 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 123 | Yes
C | A B C* | 1 | 123 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 133 | Yes
C | A B C* D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 128 | Yes
C | A B C* D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 130 | Yes
C | A B C* D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 128 | Yes
C | A B C* | 3 | 128 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 129 | Yes
C | A B C* | 1 | 129 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 136 | Yes
C | A B C* | 2 | 136 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 129 | Yes
C | A B* C D~ | 1 | 129 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 127 | Yes
C | A B* C D~ | 1 | 127 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 134 | Yes
C | A B C* D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 126 | Yes
C | A B C* | 3 | 126 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 126 | Yes
C | A B* C D~ | 2 | 126 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 136 | Yes
C | A B C* D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 130 | Yes
C | A B C* D~ | 1 | 130 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 132 | Yes
C | A B C* D~ | 1 | 132 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 139 | Yes
C | A B* C D~ | 1 | 139 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 132 | Yes
C | A B C* D~ | 3 | 132 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 133 | Yes
C | A B C* D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 129 | Yes
C | A B* C D~ | 1 | 129 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 24
Third Quartile | 26
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 146
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 139 | Yes
C | A B* C D~ | 2 | 139 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 130 | Yes
C | A B C* D~ | 2 | 130 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 139 | Yes
C | A B C* | 1 | 139 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 126 | Yes
C | A B C* | 3 | 126 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 129 | Yes
C | A B C* D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 133 | Yes
C | A B C* | 1 | 133 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 132 | Yes
C | A B C* D~ | 3 | 132 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 133 | Yes
C | A B C* D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 145 | Yes
C | A B* C | 2 | 145 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 140 | Yes
C | A B* C | 1 | 140 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 137 | Yes
C | A B C* | 2 | 137 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 139 | Yes
C | A B C* | 1 | 139 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 126 | Yes
C | A B C* | 3 | 126 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 142 | Yes
C | A B* C | 2 | 142 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 133 | Yes
C | A B C* | 1 | 133 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 137 | Yes
C | A B C* | 1 | 137 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 136 | Yes
C | A B C* | 3 | 136 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 138 | Yes
C | A B C* | 1 | 138 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 130
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.133:38751, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 145 | Yes
C | A B* C | 2 | 145 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 137 | Yes
C | A B* C D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 140 | Yes
C | A B C* | 2 | 140 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 137 | Yes
C | A B C* | 2 | 137 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 141 | Yes
C | A B C* | 1 | 141 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 131 | Yes
C | A B C* D~ | 3 | 131 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 144 | Yes
C | A B C* | 1 | 144 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 147 | Yes
C | A B* C D~ | 2 | 147 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 136 | Yes
C | A B C* | 3 | 136 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 138 | Yes
C | A B C* | 1 | 138 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 146 | Yes
C | A B C* D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 144 | Yes
C | A B C* | 2 | 144 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 139
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 145 | Yes
C | A B* C | 2 | 145 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 137 | Yes
C | A B* C D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 140 | Yes
C | A B C* | 2 | 140 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 141 | Yes
C | A B C* | 1 | 141 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 131 | Yes
C | A B C* D~ | 3 | 131 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 144 | Yes
C | A B C* | 1 | 144 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 147 | Yes
C | A B* C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 134 | Yes
C | A B C* | 1 | 134 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* D~ | 1 | 140 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 136 | Yes
C | A B C* | 3 | 136 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | No
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 146 | Yes
C | A B C* D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 144 | Yes
C | A B C* | 2 | 144 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 145 | Yes
C | A B* C D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 141
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 149 | Yes
C | A B* C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 145 | Yes
C | A B C* D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 147 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 138 | Yes
C | A B C* | 3 | 138 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* | 1 | 150 | No
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 148 | Yes
C | A B* C | 2 | 148 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 147 | Yes
C | A B* C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 148 | Yes
C | A B C* | 1 | 148 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* | 1 | 142 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 140 | Yes
C | A B C* | 1 | 145 | No
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 137 | Yes
C | A B C* D~ | 3 | 137 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 151 | Yes
C | A B C* | 1 | 151 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 145 | Yes
C | A B C* D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 140 | Yes
C | A B* C D~ | 1 | 140 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 17
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.133:38751, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 149 | Yes
C | A B* C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 145 | Yes
C | A B C* D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 141 | Yes
C | A B C* D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 147 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 138 | Yes
C | A B C* | 3 | 138 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 137 | Yes
C | A B* C | 2 | 137 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 150 | Yes
C | A B C* | 1 | 150 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 148 | Yes
C | A B* C | 2 | 148 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 138 | Yes
C | A B C* D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 148 | Yes
C | A B C* | 1 | 148 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 142 | Yes
C | A B C* | 1 | 142 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 153 | Yes
C | A B* C | 1 | 153 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 137 | Yes
C | A B C* D~ | 3 | 137 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 151 | Yes
C | A B C* | 1 | 151 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 145 | Yes
C | A B C* D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 142 | Yes
C | A B* C | 1 | 142 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 145 | Yes
C | A B C* D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 146 | Yes
C | A B C* | 2 | 146 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 147 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 138 | Yes
C | A B C* | 3 | 138 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 143 | Yes
C | A B C* | 1 | 143 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 137 | Yes
C | A B* C | 2 | 137 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 150 | Yes
C | A B C* | 1 | 150 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 154 | Yes
C | A B C* | 1 | 154 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 148 | Yes
C | A B* C | 2 | 148 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 148 | Yes
C | A B C* | 1 | 148 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 142 | Yes
C | A B C* | 1 | 142 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 153 | Yes
C | A B* C | 1 | 153 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 143 | Yes
C | A B C* | 3 | 143 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 151 | Yes
C | A B C* | 1 | 151 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 150 | Yes
C | A B* C | 1 | 150 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 21
Median | 21
Third Quartile | 22
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 158 | Yes
C | A B* C D~ | 1 | 158 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 146 | Yes
C | A B C* | 2 | 146 | Yes
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 152 | Yes
C | A B C* | 1 | 152 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 140 | Yes
C | A B C* D~ | 3 | 140 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 147 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 137 | Yes
C | A B* C | 2 | 137 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 155 | Yes
C | A B C* D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 154 | Yes
C | A B C* | 1 | 154 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 153 | Yes
C | A B* C D~ | 2 | 153 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 159 | Yes
C | A B* C D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 152 | Yes
C | A B C* D~ | 1 | 152 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 157 | Yes
C | A B* C D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 143 | Yes
C | A B C* | 3 | 143 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 147 | Yes
C | A B C* | 1 | 147 | Yes
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 156 | Yes
C | A B C* D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 155 | Yes
C | A B* C D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 26
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.131:33019, tserver@127.5.93.134:40355, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 158 | Yes
C | A B* C D~ | 1 | 158 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 151 | Yes
C | A B C* D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 152 | Yes
C | A B C* | 1 | 152 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 140 | Yes
C | A B C* D~ | 3 | 140 | Yes
D | [config not available] | | |
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 147 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 143 | Yes
C | A B* C D~ | 2 | 143 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 155 | Yes
C | A B C* D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 154 | Yes
C | A B C* | 1 | 154 | Yes
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 153 | Yes
C | A B* C D~ | 2 | 153 | Yes
D | [config not available] | | |
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 159 | Yes
C | A B* C D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 152 | Yes
C | A B C* D~ | 1 | 152 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 148 | Yes
C | A B C* D~ | 1 | 148 | Yes
D | [config not available] | | |
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 157 | Yes
C | A B* C D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 143 | Yes
C | A B C* | 3 | 143 | Yes
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 153 | Yes
C | A B C* D~ | 1 | 153 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 156 | Yes
C | A B C* D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 155 | Yes
C | A B* C D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 148 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 20
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 143
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.134:40355, tserver@127.5.93.130:43869, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 159 | Yes
C | A B* C D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
D | [config not available] | | |
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 163 | Yes
C | A B* C | 1 | 163 | Yes
D | [config not available] | | |
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 151 | Yes
C | A B C* D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 154 | Yes
C | A B C* | 1 | 154 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 158 | Yes
C | A B C* D~ | 1 | 158 | Yes
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 146 | Yes
C | A B C* | 3 | 146 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 151 | Yes
C | A B C* D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 143 | Yes
C | A B* C D~ | 2 | 143 | Yes
D | [config not available] | | |
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 155 | Yes
C | A B C* D~ | 1 | 155 | Yes
D | [config not available] | | |
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 156 | Yes
C | A B C* D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 159 | Yes
C | A B* C | 2 | 159 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 159 | Yes
C | A B* C D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 149 | Yes
C | A B C* D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 158 | Yes
C | A B C* | 1 | 158 | Yes
D | [config not available] | | |
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 153 | Yes
C | A B C* | 1 | 153 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 145 | Yes
C | A B C* D~ | 1 | 145 | Yes
D | [config not available] | | |
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
D | [config not available] | | |
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 155 | Yes
C | A B C* | 1 | 155 | Yes
D | [config not available] | | |
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 157 | Yes
C | A B* C D~ | 1 | 157 | Yes
D | [config not available] | | |
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 147 | Yes
C | A B C* D~ | 3 | 147 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 153 | Yes
C | A B C* D~ | 1 | 153 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 162 | Yes
C | A B C* | 1 | 162 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 161 | Yes
C | A B* C | 1 | 161 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable [NONVOTER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
D = 2eab0c629f5f4d6caf6d77a8198d9980
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C D~ | 1 | 148 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 21
Median | 22
Third Quartile | 25
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+--------------------+---------------
Big | d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | 32
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 142
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6736902ae7914070b2ce9c14cc9a0d6b | 127.5.93.190:36985 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.93.148:37413 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
87d5e5fccb564f69a9743f927f192070 | 127.5.93.134:40355 | HEALTHY | <none> | 11 | 0
915d43a567bf44dbaae045d9ff6c36d4 | 127.5.93.131:33019 | HEALTHY | <none> | 11 | 0
b0f058493ec44c7fb0f6465be65cf09b | 127.5.93.130:43869 | HEALTHY | <none> | 9 | 0
d633a34c7e594a35a858e82ca29625b9 | 127.5.93.133:38751 | HEALTHY | <none> | 10 | 0
2222cc707c1244e19c62a6fdb81bcc93 | 127.5.93.132:42447 | UNAVAILABLE | <none> | n/a | n/a
2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.5.93.132:42447: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.132:42447: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.5.93.129:40563: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.5.93.129:40563: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
local_ip_for_outbound_sockets | 127.5.93.130 | experimental | 127.5.93.130:43869
local_ip_for_outbound_sockets | 127.5.93.131 | experimental | 127.5.93.131:33019
local_ip_for_outbound_sockets | 127.5.93.133 | experimental | 127.5.93.133:38751
local_ip_for_outbound_sockets | 127.5.93.134 | experimental | 127.5.93.134:40355
never_fsync | true | unsafe,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.93.130:43869
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.93.131:33019
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.93.133:38751
server_dump_info_path | /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.93.134:40355
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.5.93.148:37413 | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
time_source | builtin | 127.5.93.134:40355, 127.5.93.131:33019, 127.5.93.130:43869, and 1 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.5.93.190:36985, tserver@127.5.93.130:43869, tserver@127.5.93.131:33019, and 2 other server(s)
Tablet Summary
Tablet 95662ad79bb64cfabbfd8a28d445594c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = 915d43a567bf44dbaae045d9ff6c36d4
C = b0f058493ec44c7fb0f6465be65cf09b
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 159 | Yes
C | A B* C D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet eeb1766eed8a4752ad4207cee52bf123 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet d6ab3ba8d89f437aace4599ecabb1490 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet e69f93c66579485ea31947ff7cb24fa6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 163 | Yes
C | A B* C | 1 | 163 | Yes
Tablet 61a72a79c90d407ca335335604aa146c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 151 | Yes
C | A B C* D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet e196377875a6401c80925c33730779d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 154 | Yes
C | A B C* | 1 | 154 | Yes
Tablet a9e127f655494633907fc7a733fdbf94 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 158 | Yes
C | A B C* D~ | 1 | 158 | Yes
D | [config not available] | | |
Tablet aa4df040980d4022a69625a3c23854a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 146 | Yes
C | A B C* | 3 | 146 | Yes
Tablet b949e8e50227461992887659182c4498 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 151 | Yes
C | A B C* | 1 | 151 | Yes
Tablet 7d29d5c84a1c42f4a56ca85ac941a056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 147 | Yes
C | A B* C | 2 | 147 | Yes
Tablet 561d6b28d8c34ba0ab1a6a7c317d94ef of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 160 | Yes
C | A B C* | 1 | 160 | Yes
Tablet 54a945b92ec64e59a82d1981bbfac19e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 156 | Yes
C | A B C* D~ | 1 | 156 | Yes
D | [config not available] | | |
Tablet a498828f933849cf90e1da2da475deeb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 159 | Yes
C | A B* C | 2 | 159 | Yes
Tablet 51d7d4dc02f5489d82d96ad444ebbbd4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 163 | Yes
C | A B* C | 2 | 163 | Yes
Tablet 47ffe437215a49b8bf6430d585e6891d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 149 | Yes
C | A B C* D~ | 1 | 149 | Yes
D | [config not available] | | |
Tablet 2e10d42b8e9743279084825805738340 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 158 | Yes
C | A B C* | 1 | 158 | Yes
Tablet e1a53539d67a451aa5c2166e52d28f96 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 153 | Yes
C | A B C* | 1 | 153 | Yes
Tablet 0c5069cd4ea843e1be42c086c29f686f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 150 | Yes
C | A B C* | 1 | 150 | Yes
Tablet 7e8b3b67c37844d39212180865060c46 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet d95b1095734546a089095b73ebd0b065 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 155 | Yes
C | A B C* | 1 | 155 | Yes
Tablet 205f2c275b34407bacd959718206d056 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 162 | Yes
C | A B* C | 1 | 162 | Yes
Tablet ca1037ef57c8485ca9ac5b142a6ff8c7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 147 | Yes
C | A B C* D~ | 3 | 147 | Yes
D | [config not available] | | |
Tablet 2a358caed74e43bcbf274fd5c557504a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 153 | Yes
C | A B C* D~ | 1 | 153 | Yes
D | [config not available] | | |
Tablet f34db69da7564922a4a11293aa1db952 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING [LEADER]
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 162 | Yes
C | A B C* | 1 | 162 | Yes
Tablet 69154c1b5f1c434393a4fa58a8a4bc3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2eab0c629f5f4d6caf6d77a8198d9980 (127.5.93.129:40563): TS unavailable
b0f058493ec44c7fb0f6465be65cf09b (127.5.93.130:43869): RUNNING
915d43a567bf44dbaae045d9ff6c36d4 (127.5.93.131:33019): RUNNING [LEADER]
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable [NONVOTER]
All reported replicas are:
A = 2eab0c629f5f4d6caf6d77a8198d9980
B = b0f058493ec44c7fb0f6465be65cf09b
C = 915d43a567bf44dbaae045d9ff6c36d4
D = 2222cc707c1244e19c62a6fdb81bcc93
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet eabbb051e0bc4e979572214455d681f2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 161 | Yes
C | A B* C | 1 | 161 | Yes
Tablet 4489cb8195c846b1a8c36fdd772b1b29 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
2222cc707c1244e19c62a6fdb81bcc93 (127.5.93.132:42447): TS unavailable
d633a34c7e594a35a858e82ca29625b9 (127.5.93.133:38751): RUNNING [LEADER]
87d5e5fccb564f69a9743f927f192070 (127.5.93.134:40355): RUNNING
All reported replicas are:
A = 2222cc707c1244e19c62a6fdb81bcc93
B = d633a34c7e594a35a858e82ca29625b9
C = 87d5e5fccb564f69a9743f927f192070
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 13 | 0 | 27 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 32
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 2eab0c629f5f4d6caf6d77a8198d9980 | 127.5.93.129:40563 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
E20240502 14:02:08.977181 5494 external_mini_cluster.cc:943] Process with UUID 2eab0c629f5f4d6caf6d77a8198d9980 has crashed
E20240502 14:02:08.977360 5494 external_mini_cluster.cc:943] Process with UUID 2222cc707c1244e19c62a6fdb81bcc93 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 14:02:09.660506 5494 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu with pid 5728
I20240502 14:02:09.668463 5494 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu with pid 5860
I20240502 14:02:09.675242 5494 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu with pid 6125
I20240502 14:02:09.682377 5494 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu with pid 6256
I20240502 14:02:09.688088 5494 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOVdOnA/build/debug/bin/kudu with pid 5506
2024-05-02T14:02:09Z chronyd exiting
I20240502 14:02:09.704743 5494 test_util.cc:175] -----------------------------------------------
I20240502 14:02:09.704821 5494 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOVdOnA/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714658492721238-5494-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (36975 ms)
[----------] 1 test from RaftConsensusStressITest (36975 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (36975 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240502 14:02:09.705238 5494 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 11 messages since previous log ~22 seconds ago