[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:00:17.360864 396 test_util.cc:268] Using random seed: 474105808
I20240503 08:00:17.367094 396 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:00:17.367136 396 ts_itest-base.cc:113] --------------
I20240503 08:00:17.367147 396 ts_itest-base.cc:114] 6 tablet servers
I20240503 08:00:17.367157 396 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:00:17.367165 396 ts_itest-base.cc:116] --------------
2024-05-03T08:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:00:17Z Disabled control of system clock
I20240503 08:00:17.375738 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.62:44993
--webserver_interface=127.0.99.62
--webserver_port=0
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.62:44993
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=40 with env {}
I20240503 08:00:17.540274 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 408
I20240503 08:00:17.540387 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:00:17.591787 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.1:0
--local_ip_for_outbound_sockets=127.0.99.1
--webserver_interface=127.0.99.1
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:17.797698 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 499
I20240503 08:00:17.797793 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:00:17.799785 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.2:0
--local_ip_for_outbound_sockets=127.0.99.2
--webserver_interface=127.0.99.2
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:17.974648 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 630
I20240503 08:00:17.974745 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:00:17.976716 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.3:0
--local_ip_for_outbound_sockets=127.0.99.3
--webserver_interface=127.0.99.3
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:18.131062 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 761
I20240503 08:00:18.131162 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:18.133085 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.4:0
--local_ip_for_outbound_sockets=127.0.99.4
--webserver_interface=127.0.99.4
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:18.317886 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 892
I20240503 08:00:18.317997 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:18.320112 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.5:0
--local_ip_for_outbound_sockets=127.0.99.5
--webserver_interface=127.0.99.5
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:18.494724 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 1023
I20240503 08:00:18.494822 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:00:18.496753 396 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
/tmp/dist-test-taskXVvsHM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.6:0
--local_ip_for_outbound_sockets=127.0.99.6
--webserver_interface=127.0.99.6
--webserver_port=0
--tserver_master_addrs=127.0.99.62:44993
--builtin_ntp_servers=127.0.99.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240503 08:00:18.671969 396 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu as pid 1154
I20240503 08:00:18.672051 396 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:00:18.686036 396 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 08:00:18.783900 396 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 08:00:18.785463 396 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4f9082f5c4d147e3bc12ec4d823641c1 to finish bootstrapping
I20240503 08:00:18.790690 396 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dd25402d957d4d168228a082f33cc92a to finish bootstrapping
I20240503 08:00:18.796262 396 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ba054a5fd24b4ea4a5adc5238eda63d6 to finish bootstrapping
I20240503 08:00:18.800451 396 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0b957afc99db445db0f4198081b1e2ed to finish bootstrapping
I20240503 08:00:18.805163 396 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0382071548fd4d9e838193df63f4acf9 to finish bootstrapping
I20240503 08:00:18.808965 396 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a70cc01a0b7745f98bc3ead921aea3ac to finish bootstrapping
I20240503 08:00:18.813838 396 test_util.cc:268] Using random seed: 475558794
F20240503 08:00:28.234625 1872 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-327481e521704ba9a750e92755c822c5.queue_time_us, tablet-download-pool-327481e521704ba9a750e92755c822c5.queue_time_us) (tablet-download-pool-327481e521704ba9a750e92755c822c5.run_cpu_time_us, tablet-download-pool-327481e521704ba9a750e92755c822c5.run_cpu_time_us) (tablet-download-pool-327481e521704ba9a750e92755c822c5.run_wall_time_us, tablet-download-pool-327481e521704ba9a750e92755c822c5.run_wall_time_us) (tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.queue_time_us, tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.queue_time_us) (tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.run_cpu_time_us, tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.run_cpu_time_us) (tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.run_wall_time_us, tablet-download-pool-3823d48c2019435f8aa0fb3ac54f41a3.run_wall_time_us) (tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.queue_time_us, tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.queue_time_us) (tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.run_cpu_time_us, tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.run_cpu_time_us) (tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.run_wall_time_us, tablet-download-pool-477f650fca0141d9b2cc93f9b37f8d71.run_wall_time_us) (tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.queue_time_us, tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.queue_time_us) (tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.run_cpu_time_us, tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.run_cpu_time_us) (tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.run_wall_time_us, tablet-download-pool-56134d7c8ce948e5a8506d469b60a54f.run_wall_time_us) (tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.queue_time_us, tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.queue_time_us) (tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.run_cpu_time_us, tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.run_cpu_time_us) (tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.run_wall_time_us, tablet-download-pool-6fac65a9565843f681d7741dacc7dda6.run_wall_time_us) (tablet-download-pool-7771e04e1612436db4309a0d044a6c3a.queue_time_us, tablet-download-pool-7771e04e1612436db4309a0d044a6c3a.queue_time_us) (tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.queue_time_us, tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.queue_time_us) (tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.run_cpu_time_us, tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.run_cpu_time_us) (tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.run_wall_time_us, tablet-download-pool-81b801949c3d4f6ea28d3bedfedd6c06.run_wall_time_us) (tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.queue_time_us, tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.queue_time_us) (tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.run_cpu_time_us, tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.run_cpu_time_us) (tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.run_wall_time_us, tablet-download-pool-8ebd676a4b1a4967ad28e826b432f327.run_wall_time_us) (tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.queue_time_us, tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.queue_time_us) (tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.run_cpu_time_us, tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.run_cpu_time_us) (tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.run_wall_time_us, tablet-download-pool-a0cc3570ec2d4d55af851b59b744686d.run_wall_time_us) (tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.queue_time_us, tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.queue_time_us) (tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.run_cpu_time_us, tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.run_cpu_time_us) (tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.run_wall_time_us, tablet-download-pool-b978cf1a51bc402999a7385272a58b0e.run_wall_time_us) (tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.queue_time_us, tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.queue_time_us) (tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.run_cpu_time_us, tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.run_cpu_time_us) (tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.run_wall_time_us, tablet-download-pool-c00718ff2e734c09a333e68c8a4b2ed2.run_wall_time_us) (tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.queue_time_us, tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.queue_time_us) (tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.run_cpu_time_us, tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.run_cpu_time_us) (tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.run_wall_time_us, tablet-download-pool-c1f05e8431984cdd86c18852fa494fe5.run_wall_time_us) (tablet-download-pool-c3636aaa1906448f90f67accec81e550.queue_time_us, tablet-download-pool-c3636aaa1906448f90f67accec81e550.queue_time_us) (tablet-download-pool-c3636aaa1906448f90f67accec81e550.run_cpu_time_us, tablet-download-pool-c3636aaa1906448f90f67accec81e550.run_cpu_time_us) (tablet-download-pool-c3636aaa1906448f90f67accec81e550.run_wall_time_us, tablet-download-pool-c3636aaa1906448f90f67accec81e550.run_wall_time_us) (tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.queue_time_us, tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.queue_time_us) (tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.run_cpu_time_us, tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.run_cpu_time_us) (tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.run_wall_time_us, tablet-download-pool-ca1905ad47e34bd49b0334c54d1cdd24.run_wall_time_us) (tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.queue_time_us, tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.queue_time_us) (tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.run_cpu_time_us, tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.run_cpu_time_us) (tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.run_wall_time_us, tablet-download-pool-ef1a15a9a17748a28f3d6c499023401d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4119745dcd google::LogMessage::Fail() at ??:0
@ 0x7f4119749b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f41197457cc google::LogMessage::Flush() at ??:0
@ 0x7f4119746f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f411abaf044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f411ab93f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f411ab91efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f411d74477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f411d74da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f411d7ce76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f411d7e24f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f411d7cd226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f411d7dcc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5570d8b3275c std::function<>::operator()() at ??:0
@ 0x7f411ab96e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f411ab976ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f411ab9903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5570d8b3275c std::function<>::operator()() at ??:0
@ 0x7f411ab86807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f411b3c46db start_thread at ??:0
@ 0x7f41188fc71f clone at ??:0
W20240503 08:00:28.694739 1317 meta_cache.cc:302] tablet 5c8658dd8206468480a377b583e56e03: replica dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111)
F20240503 08:00:32.033159 2426 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.queue_time_us, tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.queue_time_us) (tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.run_cpu_time_us, tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.run_cpu_time_us) (tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.run_wall_time_us, tablet-download-pool-1d3ea30092854fc495d7e0bd5e9db6df.run_wall_time_us) (tablet-download-pool-24e648eeb5454083b8642568825edb93.queue_time_us, tablet-download-pool-24e648eeb5454083b8642568825edb93.queue_time_us) (tablet-download-pool-24e648eeb5454083b8642568825edb93.run_cpu_time_us, tablet-download-pool-24e648eeb5454083b8642568825edb93.run_cpu_time_us) (tablet-download-pool-24e648eeb5454083b8642568825edb93.run_wall_time_us, tablet-download-pool-24e648eeb5454083b8642568825edb93.run_wall_time_us) (tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.queue_time_us, tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.queue_time_us) (tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.run_cpu_time_us, tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.run_cpu_time_us) (tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.run_wall_time_us, tablet-download-pool-2cc3d55898a849ecb2ab5bc78b9cd402.run_wall_time_us) (tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.queue_time_us, tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.queue_time_us) (tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.run_cpu_time_us, tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.run_cpu_time_us) (tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.run_wall_time_us, tablet-download-pool-3f31d98dceac49efba84fb02eb941a49.run_wall_time_us) (tablet-download-pool-5c8658dd8206468480a377b583e56e03.queue_time_us, tablet-download-pool-5c8658dd8206468480a377b583e56e03.queue_time_us) (tablet-download-pool-5c8658dd8206468480a377b583e56e03.run_cpu_time_us, tablet-download-pool-5c8658dd8206468480a377b583e56e03.run_cpu_time_us) (tablet-download-pool-5c8658dd8206468480a377b583e56e03.run_wall_time_us, tablet-download-pool-5c8658dd8206468480a377b583e56e03.run_wall_time_us) (tablet-download-pool-840b724de9274c44b6c19501d172b5e0.queue_time_us, tablet-download-pool-840b724de9274c44b6c19501d172b5e0.queue_time_us) (tablet-download-pool-840b724de9274c44b6c19501d172b5e0.run_cpu_time_us, tablet-download-pool-840b724de9274c44b6c19501d172b5e0.run_cpu_time_us) (tablet-download-pool-840b724de9274c44b6c19501d172b5e0.run_wall_time_us, tablet-download-pool-840b724de9274c44b6c19501d172b5e0.run_wall_time_us) (tablet-download-pool-8d92781aa1424ea5995242a864f6b108.queue_time_us, tablet-download-pool-8d92781aa1424ea5995242a864f6b108.queue_time_us) (tablet-download-pool-8d92781aa1424ea5995242a864f6b108.run_cpu_time_us, tablet-download-pool-8d92781aa1424ea5995242a864f6b108.run_cpu_time_us) (tablet-download-pool-8d92781aa1424ea5995242a864f6b108.run_wall_time_us, tablet-download-pool-8d92781aa1424ea5995242a864f6b108.run_wall_time_us) (tablet-download-pool-9cdc279e382f41eb8408bb63093c55cb.queue_time_us, tablet-download-pool-9cdc279e382f41eb8408bb63093c55cb.queue_time_us) (tablet-download-pool-af422bb22b304e148271073ccbd6ab42.queue_time_us, tablet-download-pool-af422bb22b304e148271073ccbd6ab42.queue_time_us) (tablet-download-pool-af422bb22b304e148271073ccbd6ab42.run_cpu_time_us, tablet-download-pool-af422bb22b304e148271073ccbd6ab42.run_cpu_time_us) (tablet-download-pool-af422bb22b304e148271073ccbd6ab42.run_wall_time_us, tablet-download-pool-af422bb22b304e148271073ccbd6ab42.run_wall_time_us) (tablet-download-pool-b572d7435e104e9bb3efc10504817e54.queue_time_us, tablet-download-pool-b572d7435e104e9bb3efc10504817e54.queue_time_us) (tablet-download-pool-b572d7435e104e9bb3efc10504817e54.run_cpu_time_us, tablet-download-pool-b572d7435e104e9bb3efc10504817e54.run_cpu_time_us) (tablet-download-pool-b572d7435e104e9bb3efc10504817e54.run_wall_time_us, tablet-download-pool-b572d7435e104e9bb3efc10504817e54.run_wall_time_us) (tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.queue_time_us, tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.queue_time_us) (tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.run_cpu_time_us, tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.run_cpu_time_us) (tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.run_wall_time_us, tablet-download-pool-c53bf93d5dca4093942ae4013c7040dc.run_wall_time_us) (tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.queue_time_us, tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.queue_time_us) (tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.run_cpu_time_us, tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.run_cpu_time_us) (tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.run_wall_time_us, tablet-download-pool-ddfce605c6f5405f90ff3826ce795a95.run_wall_time_us) (tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.queue_time_us, tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.queue_time_us) (tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.run_cpu_time_us, tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.run_cpu_time_us) (tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.run_wall_time_us, tablet-download-pool-e37e351977bd4aa4b1b1d4942c0613a8.run_wall_time_us) (tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.queue_time_us, tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.queue_time_us) (tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.run_cpu_time_us, tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.run_cpu_time_us) (tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.run_wall_time_us, tablet-download-pool-f1a5605ceb1143b2b3e3027ca0001750.run_wall_time_us) (tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.queue_time_us, tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.queue_time_us) (tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.run_cpu_time_us, tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.run_cpu_time_us) (tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.run_wall_time_us, tablet-download-pool-f4cf58d944db44edad7adeeacf31c879.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9f0b13bdcd google::LogMessage::Fail() at ??:0
@ 0x7f9f0b13fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9f0b13b7cc google::LogMessage::Flush() at ??:0
@ 0x7f9f0b13cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9f0c5a5044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f9f0c589f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f9f0c587efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f9f0f13a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f9f0f143a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f9f0f1c476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f9f0f1d84f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f9f0f1c3226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f9f0f1d2c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559d3b7cf75c std::function<>::operator()() at ??:0
@ 0x7f9f0c58ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9f0c58d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9f0c58f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559d3b7cf75c std::function<>::operator()() at ??:0
@ 0x7f9f0c57c807 kudu::Thread::SuperviseThread() at ??:0
Master Summary
@ 0x7f9f0cdba6db start_thread at ??:0
@ 0x7f9f0a2f271f clone at ??:0
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | HEALTHY | <none> | 0 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 6 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 14 | 0
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.4 | experimental | 127.0.99.4:43159
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.4:43159
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.4:43159, 127.0.99.1:35647, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 3 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet f1a5605ceb1143b2b3e3027ca0001750: Tablet copy aborted
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 60 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet 8ebd676a4b1a4967ad28e826b432f327 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 57 | Yes
B | A* B C | 1 | 57 | Yes
C | [config not available] | | |
Tablet 5c8658dd8206468480a377b583e56e03 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 66 | Yes
B | A B* C | 2 | 66 | Yes
C | A B C | 2 | 66 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 72 | Yes
B | A B* C | 2 | 71 | Yes
C | A B* C | 2 | 71 | Yes
Tablet af422bb22b304e148271073ccbd6ab42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 64 | Yes
B | A B* C | 2 | 64 | Yes
C | A B C | 2 | 64 | Yes
Tablet ef1a15a9a17748a28f3d6c499023401d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 72 | Yes
B | A B* C | 1 | 72 | Yes
C | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): missing [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 58 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 58 | Yes
D | [config not available] | | |
Tablet 8d92781aa1424ea5995242a864f6b108 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 61 | Yes
B | A B* C | 2 | 61 | Yes
C | A B C | 2 | 61 | Yes
Tablet ca1905ad47e34bd49b0334c54d1cdd24 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 67 | Yes
B | A* B C | 1 | 67 | Yes
C | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): missing [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 67 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): missing [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 64 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet c00718ff2e734c09a333e68c8a4b2ed2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 64 | Yes
B | A* B C | 1 | 64 | Yes
C | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | -1 | Yes
B | [config not available] | | |
C | A* B C | 3 | -1 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 56 | Yes
B | A B* C | 2 | 55 | Yes
C | A B* C | 2 | 55 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): missing [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 63 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 63 | Yes
D | [config not available] | | |
Tablet a0cc3570ec2d4d55af851b59b744686d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 64 | Yes
B | A* B C | 1 | 64 | Yes
C | [config not available] | | |
Tablet c53bf93d5dca4093942ae4013c7040dc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 71 | Yes
B | A B* C | 1 | 71 | Yes
C | A B C | 1 | 71 | Yes
Tablet 81b801949c3d4f6ea28d3bedfedd6c06 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 59 | Yes
B | A* B C | 1 | 59 | Yes
C | [config not available] | | |
Tablet 3f31d98dceac49efba84fb02eb941a49 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 68 | Yes
B | A B* C | 1 | 68 | Yes
C | A B C | 1 | 68 | Yes
Tablet 327481e521704ba9a750e92755c822c5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | A* B C | 1 | 65 | Yes
C | [config not available] | | |
Tablet 840b724de9274c44b6c19501d172b5e0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 66 | Yes
B | A B* C | 2 | 66 | Yes
C | A B C | 2 | 66 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 59 | Yes
B | A B* C | 2 | 58 | Yes
C | A B* C | 2 | 58 | Yes
Tablet b572d7435e104e9bb3efc10504817e54 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 66 | Yes
B | A B* C | 1 | 66 | Yes
C | A B C | 1 | 66 | Yes
Tablet 24e648eeb5454083b8642568825edb93 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 65 | Yes
B | A B* C | 1 | 65 | Yes
C | A B C | 1 | 65 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 65 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
Tablet f4cf58d944db44edad7adeeacf31c879 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 58 | Yes
B | A B* C | 1 | 58 | Yes
C | A B C | 1 | 58 | Yes
Tablet c3636aaa1906448f90f67accec81e550 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
All reported replicas are:
A = 0382071548fd4d9e838193df63f4acf9
B = a70cc01a0b7745f98bc3ead921aea3ac
C = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 67 | Yes
B | A* B C | 1 | 67 | Yes
C | [config not available] | | |
Tablet e37e351977bd4aa4b1b1d4942c0613a8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 65 | Yes
B | A B* C | 1 | 65 | Yes
C | A B C | 1 | 65 | Yes
Tablet 1d3ea30092854fc495d7e0bd5e9db6df of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 67 | Yes
B | A B* C | 1 | 67 | Yes
C | A B C | 1 | 67 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 71 | Yes
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet ddfce605c6f5405f90ff3826ce795a95 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 67 | Yes
B | A B* C | 1 | 67 | Yes
C | A B C | 1 | 67 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 68 | Yes
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 2cc3d55898a849ecb2ab5bc78b9cd402 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = ba054a5fd24b4ea4a5adc5238eda63d6
C = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 69 | Yes
B | A B* C | 1 | 69 | Yes
C | A B C | 1 | 69 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): RUNNING
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 2 | 57 | Yes
B | A B* C | 2 | 56 | Yes
C | A B* C | 2 | 56 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 40 | 6 | 0 | 27 | 7
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 21
Third Quartile | 22
Maximum | 29
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | 29
Small | dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
W20240503 08:00:35.470934 1317 meta_cache.cc:302] tablet f1a5605ceb1143b2b3e3027ca0001750: replica dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521) has failed: Network error: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111)
W20240503 08:00:35.502701 1317 connection.cc:535] client connection to 127.0.99.4:43159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.6:44111, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 62 | Yes
B | [config not available] | | |
C | A B C* | 2 | 62 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 73 | Yes
C | A B* C | 2 | 73 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 60 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 71 | Yes
C | A B* C | 2 | 71 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 69 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 68 | Yes
C | A B C* | 3 | 68 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 62 | Yes
B | [config not available] | | |
C | A* B C | 3 | 62 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 57 | Yes
C | A B* C | 2 | 57 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 65 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 60 | Yes
C | A B* C | 2 | 60 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 66 | Yes
C | A B C* | 2 | 66 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 66 | Yes
C | A B* C | 2 | 66 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 63 | Yes
C | A B C* | 2 | 63 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 58 | Yes
C | A B* C | 2 | 58 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 73 | Yes
C | A B* C | 2 | 73 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 60 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 71 | Yes
C | A B* C | 2 | 71 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 69 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 68 | Yes
C | A B C* | 3 | 68 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 3 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 66 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 57 | Yes
C | A B* C | 2 | 57 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 65 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 65 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 60 | Yes
C | A B* C | 2 | 60 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 66 | Yes
C | A B C* | 2 | 66 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 66 | Yes
C | A B* C | 2 | 66 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 63 | Yes
C | A B C* | 2 | 63 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 72 | Yes
C | A B* C | 2 | 72 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 58 | Yes
C | A B* C | 2 | 58 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | [config not available] | | |
C | A* B C | 1 | 65 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 75 | Yes
C | A B* C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 72 | Yes
B | [config not available] | | |
C | A* B C | 2 | 72 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 72 | Yes
C | A B C* D~ | 3 | 72 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 72 | Yes
B | [config not available] | | |
C | A* B C | 2 | 72 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 66 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 63 | Yes
C | A B* C D~ | 2 | 63 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 69 | Yes
B | [config not available] | | |
C | A* B C | 2 | 69 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 64 | Yes
C | A B* C D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 70 | Yes
C | A B C* D~ | 2 | 70 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 71 | Yes
C | A B* C D~ | 2 | 71 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 77 | Yes
C | A B* C D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 63 | Yes
C | A B* C D~ | 2 | 63 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 66 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 65 | Yes
B | [config not available] | | |
C | A* B C | 1 | 65 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 75 | Yes
C | A B* C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 72 | Yes
B | [config not available] | | |
C | A* B C | 2 | 72 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 72 | Yes
C | A B C* D~ | 3 | 72 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 72 | Yes
B | [config not available] | | |
C | A* B C | 2 | 72 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 66 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 66 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 63 | Yes
C | A B* C D~ | 2 | 63 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 69 | Yes
B | [config not available] | | |
C | A* B C | 2 | 69 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 64 | Yes
C | A B* C D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 70 | Yes
C | A B C* D~ | 2 | 70 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 71 | Yes
C | A B* C D~ | 2 | 71 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 77 | Yes
C | A B* C D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 63 | Yes
C | A B* C D~ | 2 | 63 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 70 | Yes
B | [config not available] | | |
C | A B C* | 2 | 70 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 69 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 78 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 76 | Yes
C | A B C* | 3 | 76 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 2 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 75 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 69 | Yes
B | [config not available] | | |
C | A* B C | 3 | 69 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 75 | Yes
C | A B C* | 2 | 75 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 66 | Yes
C | A B* C | 2 | 66 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 13
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 130
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 70 | Yes
B | [config not available] | | |
C | A B C* | 2 | 70 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 78 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 76 | Yes
C | A B C* | 3 | 76 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 69 | Yes
B | [config not available] | | |
C | A* B C | 3 | 69 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 75 | Yes
C | A B C* | 2 | 75 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 66 | Yes
C | A B* C | 2 | 66 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 80 | Yes
C | A B* C | 2 | 80 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 78 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 76 | Yes
C | A B C* | 3 | 76 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 75 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 67 | Yes
C | A B* C | 2 | 67 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 75 | Yes
C | A B C* | 2 | 75 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 72 | Yes
C | A B C* | 2 | 72 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 83 | Yes
C | A B* C | 2 | 83 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 66 | Yes
C | A B* C | 2 | 66 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 129
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 72 | Yes
B | [config not available] | | |
C | A* B C | 1 | 72 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 85 | Yes
C | A B* C D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 81 | Yes
B | [config not available] | | |
C | A* B C | 2 | 81 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 79 | Yes
C | A B C* D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 81 | Yes
B | [config not available] | | |
C | A* B C | 2 | 81 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 71 | Yes
C | A B* C D~ | 2 | 71 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 78 | Yes
B | [config not available] | | |
C | A* B C | 2 | 78 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 80 | Yes
C | A B C* D~ | 2 | 80 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 79 | Yes
C | A B* C D~ | 2 | 79 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 77 | Yes
C | A B C* D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 86 | Yes
C | A B* C D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 74 | Yes
B | [config not available] | | |
C | A B C* | 2 | 79 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 84 | Yes
C | A B* C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 72 | Yes
B | [config not available] | | |
C | A* B C | 1 | 72 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 85 | Yes
C | A B* C D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 81 | Yes
B | [config not available] | | |
C | A* B C | 2 | 81 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 79 | Yes
C | A B C* D~ | 3 | 79 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 81 | Yes
B | [config not available] | | |
C | A* B C | 2 | 81 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 75 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 71 | Yes
C | A B* C D~ | 2 | 71 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 78 | Yes
B | [config not available] | | |
C | A* B C | 2 | 78 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 80 | Yes
C | A B C* D~ | 2 | 80 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 79 | Yes
C | A B* C D~ | 2 | 79 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 77 | Yes
C | A B C* D~ | 2 | 77 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 86 | Yes
C | A B* C D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 79 | Yes
B | [config not available] | | |
C | A B C* | 2 | 79 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 90 | Yes
C | A B* C | 2 | 90 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 72 | Yes
B | [config not available] | | |
C | A* B C | 1 | 72 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 85 | Yes
C | A B* C D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 2 | 86 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 86 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 83 | Yes
C | A B C* | 3 | 83 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 81 | Yes
B | [config not available] | | |
C | A* B C | 2 | 81 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 80 | Yes
B | [config not available] | | |
C | A* B C | 3 | 80 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 71 | Yes
C | A B* C D~ | 2 | 71 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 74 | Yes
C | A B* C D~ | 2 | 74 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 86 | Yes
C | A B C* | 2 | 86 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 79 | Yes
C | A B* C D~ | 2 | 79 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 83 | Yes
C | A B C* | 2 | 83 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 91 | Yes
C | A B* C | 2 | 91 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 70 | Yes
C | A B* C D~ | 2 | 70 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 79 | Yes
B | [config not available] | | |
C | A B C* | 2 | 79 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 90 | Yes
C | A B* C | 2 | 90 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 77 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 90 | Yes
C | A B* C | 2 | 90 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 86 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 83 | Yes
C | A B C* | 3 | 83 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 85 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 80 | Yes
B | [config not available] | | |
C | A* B C | 3 | 80 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 77 | Yes
C | A B* C | 2 | 77 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 86 | Yes
C | A B C* | 2 | 86 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 83 | Yes
C | A B C* | 2 | 83 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 91 | Yes
C | A B* C | 2 | 91 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 86 | Yes
C | A B* C | 2 | 86 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 74 | Yes
C | A B* C | 2 | 74 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 90 | Yes
C | A B* C | 2 | 90 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 77 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 77 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 90 | Yes
C | A B* C | 2 | 90 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 86 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 83 | Yes
C | A B C* | 3 | 83 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 85 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 77 | Yes
C | A B* C | 2 | 77 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 76 | Yes
C | A B* C | 2 | 76 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 86 | Yes
C | A B C* | 2 | 86 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 83 | Yes
C | A B C* | 2 | 83 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 91 | Yes
C | A B* C | 2 | 91 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 86 | Yes
C | A B* C | 2 | 86 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 74 | Yes
C | A B* C | 2 | 74 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 129
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 97 | Yes
C | A B* C D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 83 | Yes
B | [config not available] | | |
C | A* B C | 1 | 83 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 90 | Yes
B | [config not available] | | |
C | A* B C | 2 | 90 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 88 | Yes
C | A B C* D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 95 | Yes
C | A B* C D~ | 2 | 95 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 80 | Yes
C | A B* C D~ | 2 | 80 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 84 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 84 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 97 | Yes
C | A B* C D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 83 | Yes
B | [config not available] | | |
C | A* B C | 1 | 83 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 90 | Yes
B | [config not available] | | |
C | A* B C | 2 | 90 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 85 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 88 | Yes
C | A B C* D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 95 | Yes
C | A B* C D~ | 2 | 95 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 80 | Yes
C | A B* C D~ | 2 | 80 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.6:44111, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 88 | Yes
B | [config not available] | | |
C | A B C* | 2 | 88 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 97 | Yes
C | A B* C D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 83 | Yes
B | [config not available] | | |
C | A* B C | 1 | 83 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 90 | Yes
B | [config not available] | | |
C | A* B C | 2 | 90 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 86 | Yes
B | [config not available] | | |
C | A* B C | 3 | 86 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 81 | Yes
C | A B* C D~ | 2 | 81 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 89 | Yes
B | [config not available] | | |
C | A* B C | 2 | 89 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 82 | Yes
C | A B* C D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 88 | Yes
C | A B C* D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 86 | Yes
C | A B C* D~ | 2 | 86 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C D~ | 2 | 95 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 80 | Yes
C | A B* C D~ | 2 | 80 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.3:43943, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 88 | Yes
B | [config not available] | | |
C | A B C* | 2 | 88 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 101 | Yes
C | A B* C | 2 | 101 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 86 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 95 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 95 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 92 | Yes
C | A B C* | 3 | 92 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 93 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 86 | Yes
B | [config not available] | | |
C | A* B C | 3 | 86 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 84 | Yes
C | A B* C | 2 | 84 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 94 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 94 | Yes
C | A B C* | 2 | 94 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 93 | Yes
C | A B C* | 2 | 93 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 2 | 92 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 92 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 101 | Yes
C | A B* C | 2 | 101 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 86 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 95 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 95 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 92 | Yes
C | A B C* | 3 | 92 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 93 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 86 | Yes
B | [config not available] | | |
C | A* B C | 3 | 86 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 84 | Yes
C | A B* C | 2 | 84 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 94 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 94 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 94 | Yes
C | A B C* | 2 | 94 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 93 | Yes
C | A B C* | 2 | 93 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 98 | Yes
C | A B* C | 2 | 98 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 85 | Yes
C | A B* C | 2 | 85 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 92 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 105 | Yes
C | A B* C D~ | 2 | 105 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 1 | 90 | Yes
B | [config not available] | | |
C | A* B C | 1 | 90 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 95 | Yes
C | A B C* D~ | 3 | 95 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 92 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 92 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 88 | Yes
C | A B* C D~ | 2 | 88 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 99 | Yes
B | [config not available] | | |
C | A* B C | 2 | 99 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 100 | Yes
C | A B C* D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 131
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 92 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 105 | Yes
C | A B* C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 90 | Yes
B | [config not available] | | |
C | A* B C | 1 | 90 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 95 | Yes
C | A B C* D~ | 3 | 95 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 92 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 92 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 88 | Yes
C | A B* C D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 99 | Yes
B | [config not available] | | |
C | A* B C | 2 | 99 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 100 | Yes
C | A B C* D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 96 | Yes
B | [config not available] | | |
C | A B C* | 2 | 96 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 105 | Yes
C | A B* C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 90 | Yes
B | [config not available] | | |
C | A* B C | 1 | 90 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 95 | Yes
C | A B C* D~ | 3 | 95 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 100 | Yes
B | [config not available] | | |
C | A* B C | 2 | 100 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 3 | 98 | Yes
B | [config not available] | | |
C | A* B C | 3 | 98 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 88 | Yes
C | A B* C D~ | 2 | 88 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 99 | Yes
B | [config not available] | | |
C | A* B C | 2 | 99 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 101 | Yes
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 100 | Yes
C | A B C* D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 104 | Yes
C | A B* C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 90 | Yes
C | A B* C D~ | 2 | 90 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 96 | Yes
B | [config not available] | | |
C | A B C* | 2 | 96 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 109 | Yes
C | A B* C | 2 | 109 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 95 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 104 | Yes
C | A B* C | 2 | 104 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 98 | Yes
C | A B C* | 3 | 98 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 98 | Yes
B | [config not available] | | |
C | A* B C | 3 | 98 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 93 | Yes
C | A B* C | 2 | 93 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 103 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 105 | Yes
C | A B C* | 2 | 105 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 104 | Yes
C | A B C* | 2 | 104 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 109 | Yes
C | A B* C | 2 | 109 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.3:43943, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 2 | 101 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 101 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 109 | Yes
C | A B* C | 2 | 109 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 95 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 104 | Yes
C | A B* C | 2 | 104 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 98 | Yes
C | A B C* | 3 | 98 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 98 | Yes
B | [config not available] | | |
C | A* B C | 3 | 98 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 93 | Yes
C | A B* C | 2 | 93 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 103 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 103 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 105 | Yes
C | A B C* | 2 | 105 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 104 | Yes
C | A B C* | 2 | 104 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 109 | Yes
C | A B* C | 2 | 109 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 101 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 109 | Yes
C | A B* C | 2 | 109 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 95 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 109 | No
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 104 | Yes
C | A B C* D~ | 3 | 104 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 108 | Yes
B | [config not available] | | |
C | A* B C | 2 | 108 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 103 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 93 | Yes
C | A B* C | 2 | 93 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 94 | Yes
C | A B* C | 2 | 94 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 110 | Yes
C | A B C* D~ | 2 | 110 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 113 | Yes
C | A B* C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 108 | Yes
C | A B* C | 2 | 108 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 95 | Yes
C | A B* C | 2 | 95 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 129
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 101 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 114 | Yes
C | A B* C D~ | 2 | 114 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 98 | Yes
B | [config not available] | | |
C | A* B C | 1 | 98 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 104 | Yes
C | A B C* D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 108 | Yes
B | [config not available] | | |
C | A* B C | 2 | 108 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 103 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 97 | Yes
C | A B* C D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 110 | Yes
C | A B C* D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 113 | Yes
C | A B* C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 98 | Yes
C | A B* C D~ | 2 | 98 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 105 | Yes
B | [config not available] | | |
C | A B C* | 2 | 105 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 114 | Yes
C | A B* C D~ | 2 | 114 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 98 | Yes
B | [config not available] | | |
C | A* B C | 1 | 98 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 104 | Yes
C | A B C* D~ | 3 | 104 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 108 | Yes
B | [config not available] | | |
C | A* B C | 2 | 108 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 103 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 97 | Yes
C | A B* C D~ | 2 | 97 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | [config not available] | | |
C | A* B C | 2 | 109 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 100 | Yes
C | A B* C D~ | 2 | 100 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 110 | Yes
C | A B C* D~ | 2 | 110 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 113 | Yes
C | A B* C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 98 | Yes
C | A B* C D~ | 2 | 98 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 105 | Yes
B | [config not available] | | |
C | A B C* | 2 | 105 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 113 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 107 | Yes
C | A B C* | 3 | 107 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 112 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 109 | Yes
B | [config not available] | | |
C | A* B C | 3 | 109 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 102 | Yes
C | A B* C | 2 | 102 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 113 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 114 | Yes
C | A B C* | 2 | 114 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 113 | Yes
C | A B C* | 2 | 113 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 2 | 112 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 112 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 113 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 107 | Yes
C | A B C* | 3 | 107 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 112 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 109 | Yes
B | [config not available] | | |
C | A* B C | 3 | 109 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 102 | Yes
C | A B* C | 2 | 102 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 113 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 114 | Yes
C | A B C* | 2 | 114 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 113 | Yes
C | A B C* | 2 | 113 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 112 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 113 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 113 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 107 | Yes
C | A B C* | 3 | 107 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 119 | Yes
B | [config not available] | | |
C | A* B C | 2 | 119 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 115 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 115 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 102 | Yes
C | A B* C | 2 | 102 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 120 | Yes
B | [config not available] | | |
C | A* B C | 2 | 120 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 105 | Yes
C | A B* C | 2 | 105 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | Yes
C | A B C* D~ | 2 | 117 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 113 | Yes
C | A B C* | 2 | 113 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 112 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 125 | Yes
C | A B* C D~ | 2 | 125 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 110 | Yes
B | [config not available] | | |
C | A* B C | 1 | 110 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 116 | Yes
B | [config not available] | | |
C | A* B C | 2 | 116 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 119 | Yes
B | [config not available] | | |
C | A* B C | 2 | 119 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 115 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 115 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 105 | Yes
C | A B* C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 120 | Yes
B | [config not available] | | |
C | A* B C | 2 | 120 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | Yes
C | A B C* D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 119 | Yes
C | A B C* D~ | 2 | 119 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 106 | Yes
C | A B* C D~ | 2 | 106 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 117 | Yes
B | [config not available] | | |
C | A B C* | 2 | 117 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 125 | Yes
C | A B* C D~ | 2 | 125 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 110 | Yes
B | [config not available] | | |
C | A* B C | 1 | 110 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 116 | Yes
B | [config not available] | | |
C | A* B C | 2 | 116 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 112 | Yes
C | A B C* D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 119 | Yes
B | [config not available] | | |
C | A* B C | 2 | 119 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 115 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 115 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 105 | Yes
C | A B* C D~ | 2 | 105 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 120 | Yes
B | [config not available] | | |
C | A* B C | 2 | 120 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | Yes
C | A B C* D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 119 | Yes
C | A B C* D~ | 2 | 119 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 106 | Yes
C | A B* C D~ | 2 | 106 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 117 | Yes
B | [config not available] | | |
C | A B C* | 2 | 117 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 125 | Yes
C | A B* C D~ | 2 | 125 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 117 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 123 | Yes
C | A B* C | 2 | 123 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 121 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 117 | Yes
C | A B C* | 3 | 117 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 123 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 119 | Yes
B | [config not available] | | |
C | A* B C | 3 | 119 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 125 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 125 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 107 | Yes
C | A B* C D~ | 2 | 107 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 121 | Yes
C | A B C* | 2 | 121 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 121 | Yes
C | A B* C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 129 | Yes
C | A B* C | 2 | 129 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 106 | Yes
C | A B* C D~ | 2 | 106 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 117 | Yes
B | [config not available] | | |
C | A B C* | 2 | 117 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 117 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 123 | Yes
C | A B* C | 2 | 123 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 121 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 117 | Yes
C | A B C* | 3 | 117 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 123 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 119 | Yes
B | [config not available] | | |
C | A* B C | 3 | 119 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 125 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 125 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 121 | Yes
C | A B C* | 2 | 121 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 125 | Yes
C | A B* C | 2 | 125 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 129 | Yes
C | A B* C | 2 | 129 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 110 | Yes
C | A B* C | 2 | 110 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.6:44111, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 122 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 117 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 117 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 123 | Yes
C | A B* C | 2 | 123 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 121 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 121 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 117 | Yes
C | A B C* | 3 | 117 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 123 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 123 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 3 | 122 | No
B | [config not available] | | |
C | A* B C | 3 | 119 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 130 | Yes
B | [config not available] | | |
C | A* B C | 2 | 130 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 111 | Yes
C | A B* C | 2 | 111 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 121 | Yes
C | A B C* | 2 | 121 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 125 | Yes
C | A B* C | 2 | 125 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 126 | Yes
C | A B C* | 2 | 126 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 129 | Yes
C | A B* C | 2 | 129 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 110 | Yes
C | A B* C | 2 | 110 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.6:44111, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 122 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 123 | Yes
B | [config not available] | | |
C | A* B C | 1 | 123 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 125 | Yes
B | [config not available] | | |
C | A* B C | 2 | 125 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 122 | Yes
C | A B C* D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 128 | Yes
B | [config not available] | | |
C | A* B C | 2 | 128 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 122 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 114 | Yes
C | A B* C D~ | 2 | 114 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 130 | Yes
B | [config not available] | | |
C | A* B C | 2 | 130 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 128 | Yes
C | A B C* D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 129 | Yes
C | A B C* D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 127 | Yes
B | [config not available] | | |
C | A B C* | 2 | 127 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 123 | Yes
B | [config not available] | | |
C | A* B C | 1 | 123 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 128 | Yes
C | A B* C D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 125 | Yes
B | [config not available] | | |
C | A* B C | 2 | 125 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 122 | Yes
C | A B C* D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 128 | Yes
B | [config not available] | | |
C | A* B C | 2 | 128 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 122 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 114 | Yes
C | A B* C D~ | 2 | 114 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 130 | Yes
B | [config not available] | | |
C | A* B C | 2 | 130 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 128 | Yes
C | A B C* D~ | 2 | 128 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 129 | Yes
C | A B C* D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 127 | Yes
B | [config not available] | | |
C | A B C* | 2 | 127 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 2 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 129 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 127 | Yes
C | A B C* | 3 | 127 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 132 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 127 | Yes
B | [config not available] | | |
C | A* B C | 3 | 127 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 134 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 134 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 118 | Yes
C | A B* C D~ | 2 | 118 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 133 | Yes
C | A B C* | 2 | 133 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 129 | Yes
C | A B* C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 138 | Yes
C | A B* C | 2 | 138 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 14
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 127 | Yes
B | [config not available] | | |
C | A B C* | 2 | 127 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 127 | Yes
C | A B C* | 3 | 127 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 132 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 127 | Yes
B | [config not available] | | |
C | A* B C | 3 | 127 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 134 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 134 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 122 | Yes
C | A B* C | 2 | 122 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 133 | Yes
C | A B C* | 2 | 133 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 138 | Yes
C | A B* C | 2 | 138 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 129 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 129 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 135 | Yes
C | A B* C | 2 | 135 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 129 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 127 | Yes
C | A B C* | 3 | 127 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 138 | Yes
B | [config not available] | | |
C | A* B C | 2 | 138 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 3 | 134 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 134 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 119 | Yes
C | A B* C | 2 | 119 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 139 | Yes
B | [config not available] | | |
C | A* B C | 2 | 139 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 122 | Yes
C | A B* C | 2 | 122 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 133 | Yes
C | A B C* | 2 | 133 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 138 | Yes
C | A B* C | 2 | 138 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 129 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 129 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 133 | Yes
B | [config not available] | | |
C | A* B C | 1 | 133 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 133 | Yes
B | [config not available] | | |
C | A* B C | 2 | 133 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 134 | Yes
C | A B C* D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 138 | Yes
B | [config not available] | | |
C | A* B C | 2 | 138 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 134 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 139 | Yes
B | [config not available] | | |
C | A* B C | 2 | 139 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 127 | Yes
C | A B* C D~ | 2 | 127 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 144 | Yes
C | A B* C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 14
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 135 | Yes
B | [config not available] | | |
C | A B C* | 2 | 135 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 133 | Yes
B | [config not available] | | |
C | A* B C | 1 | 133 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 133 | Yes
B | [config not available] | | |
C | A* B C | 2 | 133 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 134 | Yes
C | A B C* D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 138 | Yes
B | [config not available] | | |
C | A* B C | 2 | 138 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 134 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 139 | Yes
B | [config not available] | | |
C | A* B C | 2 | 139 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 127 | Yes
C | A B* C D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 144 | Yes
C | A B* C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 135 | Yes
B | [config not available] | | |
C | A B C* | 2 | 135 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 140 | Yes
C | A B* C D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 1 | 138 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 138 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 133 | Yes
B | [config not available] | | |
C | A* B C | 2 | 133 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 134 | Yes
C | A B C* D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 141 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 138 | Yes
B | [config not available] | | |
C | A* B C | 3 | 138 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 144 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 127 | Yes
C | A B* C D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 143 | Yes
C | A B C* | 2 | 143 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 138 | Yes
C | A B* C D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 138 | Yes
C | A B C* D~ | 2 | 138 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 150 | Yes
C | A B* C | 2 | 150 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 123 | Yes
C | A B* C D~ | 2 | 123 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 14
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.3:43943, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 135 | Yes
B | [config not available] | | |
C | A B C* | 2 | 135 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 143 | Yes
C | A B* C | 2 | 143 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 138 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 137 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 137 | Yes
C | A B C* | 3 | 137 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 141 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 138 | Yes
B | [config not available] | | |
C | A* B C | 3 | 138 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 144 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 143 | Yes
C | A B C* | 2 | 143 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 143 | Yes
C | A B C* | 2 | 143 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 150 | Yes
C | A B* C | 2 | 150 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 137 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 143 | Yes
C | A B* C | 2 | 143 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 138 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 137 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 137 | Yes
C | A B C* | 3 | 137 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 141 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 138 | Yes
B | [config not available] | | |
C | A* B C | 3 | 138 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 128 | Yes
C | A B* C | 2 | 128 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 144 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 143 | Yes
C | A B C* | 2 | 143 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 143 | Yes
C | A B C* | 2 | 143 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 150 | Yes
C | A B* C | 2 | 150 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 144 | Yes
C | A B* C | 2 | 144 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 127 | Yes
C | A B* C | 2 | 127 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 137 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 143 | Yes
C | A B* C D~ | 2 | 149 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 142 | Yes
B | [config not available] | | |
C | A* B C | 1 | 142 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 151 | Yes
C | A B* C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 143 | Yes
B | [config not available] | | |
C | A* B C | 2 | 143 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 141 | Yes
C | A B C* D~ | 3 | 141 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 145 | Yes
B | [config not available] | | |
C | A* B C | 2 | 145 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 144 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 144 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 148 | Yes
B | [config not available] | | |
C | A* B C | 2 | 148 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C D~ | 2 | 137 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 148 | Yes
C | A B C* D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 147 | Yes
C | A B C* D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 155 | Yes
C | A B* C D~ | 2 | 155 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 148 | Yes
C | A B* C D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 13
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 2 | 144 | Yes
B | [config not available] | | |
C | A B C* | 2 | 144 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 149 | Yes
C | A B* C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 142 | Yes
B | [config not available] | | |
C | A* B C | 1 | 142 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 151 | Yes
C | A B* C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 143 | Yes
B | [config not available] | | |
C | A* B C | 2 | 143 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 141 | Yes
C | A B C* D~ | 3 | 141 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 145 | Yes
B | [config not available] | | |
C | A* B C | 2 | 145 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 144 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 144 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 148 | Yes
B | [config not available] | | |
C | A* B C | 2 | 148 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 148 | Yes
C | A B C* D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 147 | Yes
C | A B* C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 147 | Yes
C | A B C* D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 155 | Yes
C | A B* C D~ | 2 | 155 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 148 | Yes
C | A B* C D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.6:44111, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 144 | Yes
B | [config not available] | | |
C | A B C* | 2 | 144 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 149 | Yes
C | A B* C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 1 | 147 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 147 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 151 | Yes
C | A B* C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 143 | Yes
B | [config not available] | | |
C | A* B C | 2 | 143 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 141 | Yes
C | A B C* D~ | 3 | 141 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 149 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 149 | Yes
B | [config not available] | | |
C | A* B C | 3 | 149 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 133 | Yes
C | A B* C D~ | 2 | 133 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 151 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 137 | Yes
C | A B* C D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 154 | Yes
C | A B C* | 2 | 154 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 147 | Yes
C | A B* C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 147 | Yes
C | A B C* D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 160 | Yes
C | A B* C | 2 | 160 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 14
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 144 | Yes
B | [config not available] | | |
C | A B C* | 2 | 144 | Yes
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 147 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 156 | Yes
C | A B* C | 2 | 156 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 147 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 145 | Yes
C | A B C* | 3 | 145 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 149 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 149 | Yes
B | [config not available] | | |
C | A* B C | 3 | 149 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 151 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 154 | Yes
C | A B C* | 2 | 154 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 153 | Yes
C | A B* C | 2 | 153 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 151 | Yes
C | A B C* | 2 | 151 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 160 | Yes
C | A B* C | 2 | 160 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 15
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 127
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.1:35647, tserver@127.0.99.5:34963, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 148 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 147 | Yes
B | [config not available] | | |
C | A* B C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 156 | Yes
C | A B* C | 2 | 156 | Yes
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 147 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 145 | Yes
C | A B C* | 3 | 145 | Yes
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 149 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 149 | Yes
B | [config not available] | | |
C | A* B C | 3 | 149 | Yes
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 151 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 154 | Yes
C | A B C* | 2 | 154 | Yes
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 153 | Yes
C | A B* C | 2 | 153 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 151 | Yes
C | A B C* | 2 | 151 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 160 | Yes
C | A B* C | 2 | 160 | Yes
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 155 | Yes
C | A B* C | 2 | 155 | Yes
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 16
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 128
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.3:43943, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 148 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 152 | Yes
C | A B* C | 2 | 152 | Yes
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 152 | Yes
B | [config not available] | | |
C | A* B C | 1 | 152 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 162 | Yes
C | A B* C D~ | 2 | 162 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 150 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 147 | Yes
D | [config not available] | | |
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 148 | Yes
C | A B C* D~ | 3 | 148 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 152 | Yes
B | [config not available] | | |
C | A* B C | 2 | 152 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 152 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 152 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 154 | Yes
B | [config not available] | | |
C | A* B C | 2 | 154 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 159 | Yes
C | A B C* D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 153 | Yes
C | A B* C | 2 | 153 | Yes
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 155 | Yes
C | A B C* D~ | 2 | 155 | Yes
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 165 | Yes
C | A B* C D~ | 2 | 165 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 158 | Yes
C | A B* C D~ | 2 | 158 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 14
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 131
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
2a67e250099340fe83d34646dc8be344 | 127.0.99.62:44993 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.99.20:42051 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
0382071548fd4d9e838193df63f4acf9 | 127.0.99.5:34963 | HEALTHY | <none> | 16 | 0
4f9082f5c4d147e3bc12ec4d823641c1 | 127.0.99.1:35647 | HEALTHY | <none> | 5 | 0
a70cc01a0b7745f98bc3ead921aea3ac | 127.0.99.6:44111 | HEALTHY | <none> | 5 | 0
ba054a5fd24b4ea4a5adc5238eda63d6 | 127.0.99.3:43943 | HEALTHY | <none> | 15 | 0
0b957afc99db445db0f4198081b1e2ed | 127.0.99.4:43159 | UNAVAILABLE | <none> | n/a | n/a
dd25402d957d4d168228a082f33cc92a | 127.0.99.2:40521 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.0.99.4:43159: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.4:43159: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.0.99.2:40521: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.0.99.2:40521: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
local_ip_for_outbound_sockets | 127.0.99.1 | experimental | 127.0.99.1:35647
local_ip_for_outbound_sockets | 127.0.99.3 | experimental | 127.0.99.3:43943
local_ip_for_outbound_sockets | 127.0.99.5 | experimental | 127.0.99.5:34963
local_ip_for_outbound_sockets | 127.0.99.6 | experimental | 127.0.99.6:44111
never_fsync | true | unsafe,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.1:35647
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.3:43943
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.5:34963
server_dump_info_path | /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.6:44111
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.0.99.20:42051 | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
time_source | builtin | 127.0.99.5:34963, 127.0.99.1:35647, 127.0.99.6:44111, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.0.99.62:44993, tserver@127.0.99.5:34963, tserver@127.0.99.1:35647, and 2 other server(s)
Tablet Summary
Tablet f1a5605ceb1143b2b3e3027ca0001750 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING [LEADER]
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 2 | 148 | Yes
B | [config not available] | | |
C | A B C* D~ | 2 | 148 | Yes
D | [config not available] | | |
Tablet bd5f479e533c4eedb423244cb665af5a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 154 | Yes
C | A B* C D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet bc324e14411848c09c10cf4e2f3b7dd5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 152 | Yes
B | [config not available] | | |
C | A* B C | 1 | 152 | Yes
Tablet 7771e04e1612436db4309a0d044a6c3a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 162 | Yes
C | A B* C D~ | 2 | 162 | Yes
D | [config not available] | | |
Tablet 0158c0ea9b7b489aaee58846fcef9acb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 150 | Yes
B | [config not available] | | |
C | A* B C | 2 | 150 | Yes
Tablet 4c3ba51ce8284fb9a2d045bc51ef4479 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 148 | Yes
C | A B C* D~ | 3 | 148 | Yes
D | [config not available] | | |
Tablet 7377feb69efa42dab12752743af779a5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 152 | Yes
B | [config not available] | | |
C | A* B C | 2 | 152 | Yes
Tablet 0c1511f2e9b5478095c9208d3e5c484f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable [NONVOTER]
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
D = 0b957afc99db445db0f4198081b1e2ed
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 152 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 152 | Yes
D | [config not available] | | |
Tablet b978cf1a51bc402999a7385272a58b0e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 142 | Yes
C | A B* C D~ | 2 | 142 | Yes
D | [config not available] | | |
Tablet 9cdc279e382f41eb8408bb63093c55cb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4f9082f5c4d147e3bc12ec4d823641c1 (127.0.99.1:35647): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable
ba054a5fd24b4ea4a5adc5238eda63d6 (127.0.99.3:43943): RUNNING
All reported replicas are:
A = 4f9082f5c4d147e3bc12ec4d823641c1
B = dd25402d957d4d168228a082f33cc92a
C = ba054a5fd24b4ea4a5adc5238eda63d6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 154 | Yes
B | [config not available] | | |
C | A* B C | 2 | 154 | Yes
Tablet 477f650fca0141d9b2cc93f9b37f8d71 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 144 | Yes
C | A B* C D~ | 2 | 144 | Yes
D | [config not available] | | |
Tablet 3823d48c2019435f8aa0fb3ac54f41a3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 159 | Yes
C | A B C* D~ | 2 | 159 | Yes
D | [config not available] | | |
Tablet 9109d6d18c944d0bbc040dcfbd0b61e1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 156 | Yes
C | A B* C D~ | 2 | 156 | Yes
D | [config not available] | | |
Tablet 6fac65a9565843f681d7741dacc7dda6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING [LEADER]
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 155 | Yes
C | A B C* D~ | 2 | 155 | Yes
D | [config not available] | | |
Tablet c1f05e8431984cdd86c18852fa494fe5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 165 | Yes
C | A B* C D~ | 2 | 165 | Yes
D | [config not available] | | |
Tablet 56134d7c8ce948e5a8506d469b60a54f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 158 | Yes
C | A B* C D~ | 2 | 158 | Yes
D | [config not available] | | |
Tablet f53d2b49c4544b8c9d10a7896d04410a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0b957afc99db445db0f4198081b1e2ed (127.0.99.4:43159): TS unavailable
0382071548fd4d9e838193df63f4acf9 (127.0.99.5:34963): RUNNING [LEADER]
a70cc01a0b7745f98bc3ead921aea3ac (127.0.99.6:44111): RUNNING
dd25402d957d4d168228a082f33cc92a (127.0.99.2:40521): TS unavailable [NONVOTER]
All reported replicas are:
A = 0b957afc99db445db0f4198081b1e2ed
B = 0382071548fd4d9e838193df63f4acf9
C = a70cc01a0b7745f98bc3ead921aea3ac
D = dd25402d957d4d168228a082f33cc92a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 142 | Yes
C | A B* C D~ | 2 | 142 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 23 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 17
Median | 22
Third Quartile | 29
Maximum | 34
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
E20240503 08:00:55.927088 396 external_mini_cluster.cc:943] Process with UUID dd25402d957d4d168228a082f33cc92a has crashed
E20240503 08:00:55.927173 396 external_mini_cluster.cc:943] Process with UUID 0b957afc99db445db0f4198081b1e2ed has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 08:00:56.700238 396 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu with pid 499
I20240503 08:00:56.710815 396 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu with pid 761
I20240503 08:00:56.719394 396 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu with pid 1023
I20240503 08:00:56.725953 396 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu with pid 1154
I20240503 08:00:56.732285 396 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXVvsHM/build/debug/bin/kudu with pid 408
2024-05-03T08:00:56Z chronyd exiting
I20240503 08:00:56.747925 396 test_util.cc:175] -----------------------------------------------
I20240503 08:00:56.747967 396 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskXVvsHM/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714723217350060-396-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (39388 ms)
[----------] 1 test from RaftConsensusStressITest (39389 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (39389 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240503 08:00:56.748422 396 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 3 messages since previous log ~21 seconds ago